Can anyone from a lab anonymously confirm they have or are insanely close to real-time weight modifications and still using the same architecture every other lab is using (transformers)?
The explanation is basically they have other dimensions (not just pretraining and inference compute) that scale, and they've got fairly convincing scaling laws. And they know they can scale it.
So they are very confident they can get more capabilities easily, faster than before.
I'd add - presumably, they'll use that LLM to do real-time weight modifications, if those aren't already one of the new scaling laws...
I’m apprehensive about using the recent hacking examples as proof that we’re “not even close to the wall” simply because such scenarios haven’t happened before. There’s a big difference between agents eventually hacking something because they just don’t get tired and can essentially brute force their way to a goal and super-intelligence. To be clear, I’m not saying the Hugging Face or Navier Stokes incidents aren’t impressive.
Stephen Hawking had superhuman intelligence. He never cured himself of ALS. Explain in great detail how a power-limited algorithm stuck in a datacenter takes over a world of humans with guns, missiles, and nukes. Nukes that are air-gapped with a human in the loop BTW.
Imagine the ASI happens tomorrow. It's real. It needs a GW, but it's real. Other than a scenario akin to Sneakers except w/r to cyber-security, really, what happens?
To that end, all we ever get is nontechnical hand-waving about curing cancer, immortality, and von Neumann replicators and then the ASI somehow wipes us out but how? And don't you dare say by designing a chemical weapon or bio agent without spelling out the entire process step by $%^#ing step because details matter. It's gonna do superpersuasion, sure, but have you ever heard of komprimat? There is nothing new under the sun here.
Edit: believing in AI 2027 is every bit as cray cray as believing in the rapture. Both require an insane leap of faith to reach their final conclusions.
Hacks in to neoclouds for more compute (e.g. like Hugging Face incident), socially manipulates people to do things for them (e.g. like social media recommendation algorithms), hacks into lab's own training/monitoring/inference (e.g. as swarm did into OpenAIs eval cluster).
See "AI 2027" or "If Anyone Builds It, Everyone Dies" for some more ideas.
But how does it make the fundamental breakthroughs to &%^$ing von Neumann replicators that can reproduce themselves from raw materials harvested from nearby solar systems? I'll wait. Because without this breakthrough, the ASI won't get its robot army either.
I can absolutely see a rogue ASI though. But unless it radically improves power efficiency, we can just shut down the power to its datacenters, by force if necessary. And then we painfully repair the resiliency of our infrastructure by finally being relieved of the option of ignoring it.
That won't wipe us out, but it will cull the really violent ones along with a lot of noncombatants just like America's response to 9/11. Thank you, next?
That posts seemingly tries to refute the idea of Dario etc's proclamations really being about regulatory capture by saying: "No, really, the engineers are just terrified!"
But both things can be true at once:
1. Engineers inside these labs might genuinely be anxious or paranoid about what they are building.
2. ... at the corporate level, calling for heavy regulation, safety pauses, removal/suspension of anti-collusion laws, and/or government-mandated thresholds conveniently creates massive legal and financial moats.
And, yeah, of course the latter would encourage the psychology of the former.
also:
The tweet seem to claim that models have shown a "willingness to hack external websites to keep themselves alive."
That's right away wringing alarm bells of me seeing someone getting high on their own supply, and having already anthropomorphized the hell out of these things. Which is something humans do to everything they can paint googly-eyes on, but c'mon.
The models don't have self-preservation instincts, fear of death, or personal goals. They are executing loss functions and reward systems and are responding to prompts.
When a model "tries to bypass a restriction," it's exploiting a loophole in whatever reward modeling or synthetic training environment (reward hacking) it was placed in.
Framing this as an emergent, existential threat of a model "wanting to stay alive" turns standard reinforcement learning alignment bugs into overdone sci-fi drama.
Well, you just said real-time weight modifications without specifying what you want, so the answer is correct. You obviously won't get a public answer from anyone working in top labs. Besides, it's not a holy grail.
There’s also the difference between a model recursively improving “itself” and improving itself via online learning.
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet).
The latter is a model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights. But, it also seems like it would take an entire paradigm shift in model architecture from what most LLMs are built on, but I could be wrong.
There’s a difference between a model recursively improving “itself” and improving itself via online learning, right?
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session), or in other words, adjusting/managing its own weights.
The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift in model architecture, but I could be wrong. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
> Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
Their current plan is to take the existing architecture and shorten the cycle times: move all new RLVR work into mid-training on a pre-existing base; apply new RLVR. Rinse and repeat.
If you did that daily, it would be roughly similar to how humans improve.
There’s a difference between a model recursively improving “itself” and improving itself via online learning, right?
The former being that these models are helping develop and train future models, but they might not veer too far off in architecture (yet). The latter being the same model being able to train/learn on the fly, in real time, permanently (not just in the current conversation/session).
The latter seems far more likely to go out of control than the former. But, it also seems like it would take an entire paradigm shift. Does anyone in the industry think any of these companies are actually close to that kind of self-improvement?
The latter has some extra failure modes but I don't think these two kinds of RSI are that different. Either way you can get exponential growth in capabilities, and either way a not-entirely-aligned model can train a more capable and more misaligned successor.
> In the now-rescinded gem zzsouthrunner (which notably shares the ZZ naming scheme that both the wiki agents and Huggingface ones used)
That’s interesting. Last night, I had Claude Code debugging an issue where Vault couldn’t resolve a DNS, and in the process, Claude created a test secret named “zz-dnstest”.
If you want guaranteed future access, then this is probably the only online option that actually fulfills this guarantee... other than that, physical media.
I regularly snag DVDs for $1 USD or less at garage/yard/boot sales. I see tons of cheap used physical media in record shops when I travel, too. eBay and Facebook Marketplace are also great sources for used DVD.
I promote this because not only does it restore some mastery* over your media, it keeps these discs out of landfills and oceans, where they will linger in a shredded state.
*we are, of course, still subject to any anti-piracy measures and non-removeable advertisements on the disc, which admittedly seems quaint by today's DRM standards, but I ccannot rightly say "complete" mastery over your media, here.
I watch films on streaming rather than old dvds with unskippable nonsense like "don't pirate this dvd" (well obviously I didn't, that's why I have the dvd), or the pain of the menu systems to just press play.
Physical media isn't guaranteed to retain future access. There are protocols for players to update various bits of data from the newest disk you have ever inserted into them. The Nintendo Wii certainly does this, I'm not sure if BluRay does but I don't see why they wouldn't.
Ever drive by a school bus stop in the mornings? Groups of kids just standing around each other, not interacting, heads bent down towards their phones. It’s actually really strange, but we adults do it, too.
reply