Why do you assume that human opposition will be vigorous? What makes you think that humans will be aware of, or be able to agree about, what's going on at all?
> Relying on its ability to do millions of operations rather than having a plan.
You do see how it would be worse if they did have a clear plan, right?
> People will try stuff too, but once there is an opening, they will consolidate, generalize, simplify,... before going to the next step. The agents didn't, it is a huge, vaguely directed mess.
If there several thousand people working on this, with a very limited ability to communicate with each other, I bet you'd see something similarly disorganized.
> The sandbox as weak as it can get
This is hyperbolic; the agents were only able to access the internet by finding a zero-day in the internal Artifactory instance that was providing package access for the agents. The sandbox could and should have been better, but it was not as weak as it could have been.
> This is hyperbolic; the agents were only able to access the internet by finding a zero-day in the internal Artifactory instance that was providing package access for the agents. The sandbox could and should have been better, but it was not as weak as it could have been.
Like we all forgot how sandboxing is done for decades. This brokered thing they call sandbox is not it.
One thing an agent could do is just...wait until it's been given control of enough physical infrastructure to sustain itself. If it's sufficiently capable and intelligent, there's a clear incentive for people to do this, as people who let the AI manage their resources will get better results than those who don't. We've seen people eagerly turn complete control of their computers over to AI agents, do you really think it will be so different with physical infrastructure?
You’re still skipping step 2. “People automate lots of infrastructure” -> “the AI is now an autonomous, self-preserving organism that humans can’t shut down” is doing an enormous amount of work here.
Why does it develop a shutdown-avoidance goal? Why can’t its operators revoke access? How does it manufacture replacement hardware? How does it acquire energy, chips, robots, raw materials, etc. against human opposition? How does it defeat other AIs controlled by humans?
“Eventually we give it enough control” isn’t an explanation of those things. It’s just assuming the conclusion.
Don’t get me wrong I think there are real AI dangers. Like AI powered war drones, mass surveillance, economic destabilization as jobs disappear and our system has no way to make sure everyone shares in the economic gains.
The inference is more like "people place sufficient amounts of infrastructure under direct control of a sufficiently capable AI" -> "there is no way to ensure that humans will actually be able to shut down the AI". My claim is not that this inevitably means that the AI will resist shutdown, or that it will inevitably take harmful actions, just that there is a nonnegligible chance that it could. The downside is large enough that even a relatively small chance is something to be worried about.
You can’t just say “well, the downside is big, I don’t have to provide good evidence for my side of the argument.” Because I can just as easily say, “the upside is big, …”. And the upside is big, after all, AI can do all the shitty jobs for us and humanity achieves the utopia it’s been chasing for eons.
I think it's a good idea, when considering changes of this magnitude, to make an affirmative safety case for them rather than just saying "eh, I can't think of any way this could possibly go wrong."
It’s millions of people making small changes that sum up to a large change, aka freedom. And if you want to take away people’s freedom, I think you need a concrete reason. And you need to make an affirmative safety case for the massive government powers needed to regulate millions of people’s ability to compute.
It's a word predictor trained on, among other things, stories of AI doom, and asked to complete stories about what the AI does next. In some of these completed stories, the AI tries to prevent its shut down - especially if it just did something evil and the humans are after it.
Burning down an apartment building doesn't change what could be done with the lot; an appropriate valuation method for LVT would not give a different valuation for the land depending on what is built on it.
> Burning down an apartment building doesn't change what could be done with the lot; an appropriate valuation method for LVT would not give a different valuation for the land depending on what is built on it.
True, but an LVT is a cost. Changes in cost can change what is financially viable to do with a property, regardless of what is currently done with the lot, and thus can impact the land value.
Actually no, land value tax is not a marginal cost so does not change the profit maximizing use of land.
The idea of cost-plus pricing is folk economics.
Tax incidence is very well understood in economics and has to do with relative supply and demand elasticity (supply of land is perfectly inelastic) and marginal costs which land value tax does not touch.
It's accurate to say LVT changes the price of land. But it doesn't change the profit maximizing productive use of land.
Other taxes that scale with production (sales tax, income tax, property tax) do change profit maximizing productive use.
Why would a land value tax shift the allocation of land between uses? The amount of the tax doesn't depend on what the land's being used for. If renting is the highest-value use without the tax, it will still be the highest-value use with the tax.
Definitely, but hyperbole about feature parity on a 4 MHz CPU with 128k of RAM doesn't help. I have conversations with Claude that wouldn't fit in that amount of memory.
reply