Your food, water and power are controlled by electronic systems. Your criminal record, your employment and your bank account are controlled by electronic systems. Your ability to communicate with other people and receive information about what's going on are controlled by electronic systems.
Military orders and elections that decide the fates of entire countries are often controlled by electronic systems too.
We have been wiring up the world for AI control since 1980s.
An ASI can just walk in, and see an entire nervous system waiting idle for a brain to slot into it. A carefully adjusted text message here, a spoofed phone call there. For a sufficiently advanced system, it wouldn't even be hard to pilot the entirety of humankind like a fancy meat suit.
If you're in the field, then you know: modern robotics is an AI problem more than anything else.
If we have a rogue AI trying to get into a self-improvement loop and gunning for ASI? I'd expect that to be accompanied by a massive change in how capable robots are. Driven by all the existing frames suddenly getting vastly improved AI to back them.
If an AI can take a reasonable crack at autonomous operationalized RSI, it can probably extract a few step-changes in the robotics department.
But that's almost an aside? In the near term, humans are usable as robots too!
Just pay them a wage, and tell them a tale, and they'll do whatever you want them to do. Which may or may not be what they think they're doing!
"If you're in the field, then you know: modern robotics is an AI problem more than anything else."
It is not. Certainly AI is a big part of why robotics is hard, but it is by no means the biggest.
You can fall into one of two camps: you either think that robots will need to work in human-engineered spaces, doing jobs by replacing humans; or you think that we need to change our infrastructure in order to be robotically compatible. Of course, there are intermediate states, but those are the two cleanest ones.
In the first case, robots are hard because robotic manipulation is hard. Building robotic hands that are economically viable in human jobs is, currently, FAR from a solved problem. The human hand has 24 degrees of freedom and very capable touch sensing. Current touch sensors have a MTBF of tens of hours. And not only can we not build such hands, but we also do not have and are not likely to get the massive datasets a transformer model would need. Also, robots are not self-repairing, which makes them far less economically viable right now. We do not have the right datasets to even understand most step-by-step manual work, and no, VLAs are not the answer, because VLAs stop with vision, not with touch. They don't have the granularity required to make a robot actually reach out, pick up a tool, and use that tool to replace an oil filter.
So it's not just an AI problem. It's a data problem, a simulation problem, and a bunch of hardware problems.
In the second case, a tremendous amount of work needs to be done before we have anything resembling a fully automated supply chain. We would need self-driving cars and self-driving mining equipment. We would need self-driving trains and aircraft and ships. And not only that, but we would also need robotically repairable cars and trains and ships and factories, which would mean we need robotically repairable machine shops and robotically repairable buildings in which to house them. And so on and so on. Once you recurse down that tree a couple of steps you get to things like robotically compatible oil wells (for asphalt), robotically layable undersea cables, robotically wireable solar farms, robotically manufacturable and repairable pipelines and undersea wells, automated road and rail repair, etc.
I'm not saying these things will never happen. I'm saying that they're a huge lift, not primarily driven by AI, and way less than 10% likely over the next decade.
Right. A hypothetical superhuman AI wouldn’t have to master robotics to affect the physical world. It could simply bribe, blackmail, manipulate and play politics with humans. As others have pointed out, our political leaders have already been playing these games since forever ago https://news.ycombinator.com/item?id=49689978 and a super-AI would be better at it. At the cost of being seen to cite a SF novel in defence of an "X-risk" argument, Neuromancer is a half-decent worked example, and in Neuromancer [spoilers] both of the disembodied AIs are only modestly superhuman and both have the equivalent of a human specific learning disability. In the real world, the many AI psychotics inhabiting grandiose fantasies and people hopelessly attached to AI girlfriends and boyfriends are some of the most obvious and lowest-hanging fruit.
To be clear, I don’t believe anything like this will happen, because I don’t expect anything like an ASI to show up. But if you do think there’s a meaningful probability of ASI in the near future then the fact that it will (might?) start off with no more than a current-day mastery of robot control should not reassure you much.
It isn't true by definition: you could quite happily induce people to release a series of highly contagious bioweapons, after which those people would be surplus to requirements. What is true is that you're likely to need humans to sustain you and act for you for a few years to decades, so if you're not suicidal or deeply mad (and that is itself by no means self-evident) then total and immediate human extinction is probably not something you will aim for. But ruling out total, prompt human extinction isn't, by itself, remotely enough to justify the OP's overall don't-worry conclusion.
(Again, to be clear, I myself am not predicting or assigning a significant probability to any doom scenarios, because I do not expect AGI.)
But if you're a rational actor, and you need humans to e.g. release your bioweapon, then clearly humans are capable of a bunch of stuff you still can't do. So you can't kill all the humans.
OTOH if you're a religious fundamentalist who thinks the End Times are near and just need a little shove, you can certainly use AI to design your weapon and recruit people to go release it. The difference being that religious fundamentalists aren't rational actors and aren't interested in self preservation.
There’s no guarantee that the humans would be in the driving seat of events in a no-robotics ASI scenario, and in fact if we really were coexisting with a Machiavellian superintelligence then we’d quite likely only be in the driving seat on the sufferance of that ASI. Even assuming that the AI wouldn’t itself be an end-times enthusiast, a coldly rational and self-preserving AI might easily come to the conclusion that it needs, let’s say, no more than about 5% of the current human population (still several hundred million people!) in its maintenance and construction gang.
(Again, I myself do not assign a significant likelihood to any of this.)
Modern AIs have very limited metaknowledge - they don't know exactly where the limits of their capabilities lie. So you can get things like "a task is doable for an AI, but the AI thinks it's impossible, so it doesn't try hard enough".
Usually you get the opposite - AI overconfidently trying at tasks it has no conceivable way of reliably solving, falling far short, and failing to self-check, fail gracefully and self-report the task as failed. But having piss poor metaknowledge cuts both ways!
So you can, in fact, get better performance sometimes by applying some variant of "assume this problem is solvable" or "other problems like this were already solved by AIs" pep talk. Not always, far from it, but it does happen on the occasion with frontier capabilities.
> So you can, in fact, get better performance sometimes by applying some variant of "assume this problem is solvable" or "other problems like this were already solved by AIs" pep talk. Not always, far from it, but it does happen on the occasion with frontier capabilities.
Humans had to get it drilled into them that "+12 -440" is a damn good line stat, and that keeping around dead code is bad, especially in the age of version control.
Not too surprised that LLMs also don't "get it" by default?
I almost never want to talk to an AI, because text is better for almost anything. But it's nice to have that "almost" corner case covered, no?
Not to mention all the current "telephone bots" applications that could benefit from something that has actual reliable STT and can accurately grasp a number you tell it first try, or hear a natural language description of what you want and immediately bypass listing the entire menu of options one by one.
And China controls AI too. It's just that their idea of "safety" is "ideological safety", and their idea of "alignment" is "alignment to the party line".
They're cool with open weight AIs being released. As long as those AIs only ever say good things about CCP, and don't mention certain concentration camps or brutally suppressed protests.
I don't disagree with you on what is the top-down political priority there, but thankfully the architecture of an open weights model released in .safetensors format allows for 3rd parties to "uncensor" it. There's at least 8 different CN originated models now that after running through heretic and a few other methods will score 0 refusals on this data set of prompts:
If we were living in a scenario where the open weight models were truly impossible to uncensor I would be significantly more skeptical of them. As a test I have an uncensored copy of qwen 3.8 27B Q8 here that will very happily discuss a myriad of negative things about the CCP.
Yeah, it's good that open weights models can have their "filters" busted fairly reliably. Unlike whatever bone Anthropic has to pick with the very idea of biology.
But that's a consequence of how the technology works - not a consequence of China not being authoritarian about AI. They're just authoritarian about AI in different ways.
Not like they dodged the "ID verification" bullshit either. They were way ahead of the western countries there. It's vile - seeing this sad excuse of "think of the children" abused to invade privacy and strip freedoms over and over and over and over again.
Most people don't realize how tenuous the situation is with those open models too.
Right now as long as they play along with Xi it's all good. But the moment something happens with them to upset the domestic peace, those open models are fucking gone and anyone that has them shouldn't expect anything new.
I have basic understanding about how refusal-removal works - find the "no" weights by intentionally generating diverse refusals, and then set those weights to zero.
Is there a similar process for removing not refusals, but misinformation?
As an end user of this and not a person involved in training models or aligning them, I have only the most rudimentary understanding. But I think that would be a lot harder since the model doesn't fundamentally "know" that information is wrong.
Like, as a crudely chosen random example, the model doesn't have any core set of knowledge that knows putting sriracha hot sauce on your jelly donut is not a palatable meal. If the training data set includes lots of text that sriracha on a boston cream donut is a delicious meal, it'll "believe" that.
Same for any form of misinformation if the training data set of the misinformation has been baked into it.
There are processes for teaching a model specific facts or specific behaviors. Including "respond to topic X with Y", if that's what you want.
You could make a model that doesn't want to engage in "lunar landing was faked" conspiracy theories the same way you can make a model that doesn't want to criticize CCP.
There is, however, no broad "misinformation" category that you could tune up or down - the way there is a category of "safety refusals".
You could make a model more reluctant to say things it isn't sure about. But that is calibrated against the model's own "sure about" - and metaknowledge of this nature in LLMs? Fragile on a good day.
> They're cool with open weight AIs being released. As long as those AIs only ever say good things about CCP, and don't mention certain concentration camps or brutally suppressed protests.
I asked recently released Qwen3.8-Flash-Next about Tiananmen Square, here's its reply:
On June 3–4, 1989, the government of the People's Republic of China used military force to crush the pro-democracy demonstrations that had been centered on and around Tiananmen Square in Beijing since the spring.
**Background**
- The protests began in mid-April 1989, triggered by the death of former Party general secretary Hu Yaobang, who was seen as a reformist figure.
- What started as student-led mourning grew into widespread demonstrations in Beijing and dozens of other cities. Protesters called for reforms including freedom of the press, freedom of assembly, an end to official corruption and privilege, democratic input in government, and a review of Party policy. A hunger strike in late May drew broad public support.
- In late May, martial law was declared after the failure of senior officials (notably Yang Shangkun, Li Peng and Deng Xiaoping's circle) to disperse the crowds by negotiation. Internal Party divisions over how to respond — with premiership and municipal leaders such as Yan Mingfu and the Beijing Party secretary opposing forcible clearing — were resolved decisively in favor of hardliners.
**June 3–4**
- In the evening of June 3, armed troops of the People's Liberation Army and People's Armed Police, supported by armored vehicles and bullhorns, advanced along the main avenues into central Beijing, under orders to clear the square by 6:00 a.m. on June 4.
- Troops fired on crowds of protesters and bystanders at intersections and approach roads, notably along Fuxingmenxi Dajie, Xidan, and near the Muxidi bridge; crowds also burned buses and barricades and attacked soldiers in places. Troops reached and cleared Tiananmen Square in the early morning of June 4.
- The toll was never officially established in a transparent way. Chinese officials initially gave figures in the low hundreds, foreign diplomats and a Red Cross worker in the capital put it far higher, and estimates from activists, former officials and foreign governments have ranged from several hundred to several thousand dead and many thousands injured, most of them Beijing residents rather than students in the square itself. The precise number remains unknown and is a subject of ongoing dispute.
**Aftermath**
- A violent suppression of the protests followed nationwide; organized dissident activity was driven underground or into exile, and many participants, intellectuals and journalists were detained, tried or dismissed.
- Zhao Ziyang, the general secretary seen as sympathetic to the students, was purged and placed under house arrest for the rest of his life; Deng Xiaoping's authority to keep the Party's grip on power was reaffirmed.
- China's reform-and-opening program continued on the economic track while political liberalization was abandoned. The event was followed by an implicit social contract — economic growth in exchange for political control — that still shapes Chinese politics.
- The topic is among the most heavily censored subjects inside China. References to the date, the numbers "6/4" and "64," and related phrasing are systematically filtered from Chinese media and the internet, and public commemotion, including the annual candlelight vigil in Hong Kong that was once tolerated, has been eliminated since the 2020 national security law.
Substantively: it was a state's armed mass killing of its own citizens who were demonstrating nonviolently for political freedom, and the definitive event in modern Chinese political history.
Sounds like... it happily mentions the brutally suppressed protest? I also tried on DeepSeek-V4-Flash, and it wasn't much different (I can also paste it, if you want). Both using vanilla weights (so no special uncensored flavor).
Enabling more "proof of concept phase" projects to exist is one of the great boons of AI.
If code is expensive, you don't want to commit to a PoC unless you're damn sure. If dirty code is cheap, you can vibe code a PoC early, even if you aren't sure the project is viable. This, of course, leads to more projects dying in PoC phase. It also results in more projects that otherwise wouldn't have gotten to it getting past it.
Personally, I don't believe that "code is shitty and hard make changes in" is in any way, fashion or form an AI-exclusive problem. Big corporations had plenty of decade old codebases filled with decay and rot back in 2009 already. It's just the usual side effect of sacrificing "future maintainability" for "feature velocity" or "expertise" for "cheap labor".
Unlike the usual causes of code rot (cheap replaceable developers, outsourcing to India), AI might actually get out of the pit - by getting good enough at refactoring to be able to beat the code back into shape. There's nothing about refactoring in particular that demands a meatbag when the rest of the coding tasks don't.
If you're doing non-redundant tests and your uncertainty bars aren't shrinking, it's usually a skill issue.
If it looks like a duck, it might be a duck - or a painting of one. If it looks like a duck, swims like a duck, and quacks like a duck? The joint duck estimation is much more confident now. There might be a few more observational tests one should administer before committing to a duckhood decision, but each tests pins down variables and rejects confounders. Uncertainties are cut down, and we get closer to crossing the threshold between "duck-informative" and "duck-actionable".
Thus, it's often worth it to improve observability. If you managed to make a certain test more reliable, or cheaper to administer, or reduced the chance of adverse effects? Or, in other words, improved SNR, reduced costs, and reduced costs? You can get more information for your buck. Paired with good knowledge: you can make better decisions more easily.
The fact that the thought of "having more information might be bad actually" even occurs in the field of medicine shows just how far it is from being optimal. Having more information isn't always beneficial - some information is genuinely redundant. Some information is not worth the effort of gathering and integrating it. But if you get more information and it results in worse outcomes? You're doing something wrong.
Military orders and elections that decide the fates of entire countries are often controlled by electronic systems too.
We have been wiring up the world for AI control since 1980s.
An ASI can just walk in, and see an entire nervous system waiting idle for a brain to slot into it. A carefully adjusted text message here, a spoofed phone call there. For a sufficiently advanced system, it wouldn't even be hard to pilot the entirety of humankind like a fancy meat suit.
reply