I was a biophysicist and experimental biologist for decades, I designed viral vectors and cell therapies, etc. What protects us here isn't our ignorance vs an "ASI".
It's the fact that the complexity of molecular physics scales exponentially in particle number. No amount of thinking is going to punch through that. You've got to do experiments. The slow loop through reality is not optional, and in the case of "superviruses" would require a lot of iterative experimental development in humans.
(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
>(And please knock off the bio infohazard act - I routinely get asked about scenarios that AIxBio safety people cook up and they always involve some howler misunderstanding of basic facts in biology or medicine. Just talk about an idea if you have it.)
We're already doing those experiments. Tailor made mRNA vaccines targeted to one's own specific cancer mutations can be bought right now. The techniques are getting more sophisticated and more targeted every year. Our ability to predict what happens at these levels is improving by leaps and bounds too thanks to AI like AlphaFold. Anyone can be reasonably confident that no AI or teen can do this now or in the next few years, but are you really so confident what might be possible in 10 years?
I worked in immuno-oncology: cancer vaccines work in melanoma where many things work because of the neoantigen abundance, they've generally been very mixed in efficacy. And even BionTech's BNT111 failed in melanoma! We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.
Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
Again, most people just have no idea how hard - fundamentally hard - molecular physics is to predict, and how necessary experiments are for any development of biological systems.
> We have hope for these approaches but the reality of this stuff is way more nuanced than you think it is.
Sure, everything has more nuance. The point is this stuff is available now; this isn't some future sci-fi, it's only going to get better, it's not the only research on gene targeting, and AI is starting to help with this research. By the time AGI is actually here, consider the breadth of knowledge and capabilities that will be at its disposal.
> Alphafold can't reliably predict thermal energy landscapes or make functional predictions - and how could it? It wasn't trained on anything that could capture structure - function relationships.
If your point is that the only reason an AI like AlphaFold can't make functional predictions is that we haven't trained an AI to do that, then unless you're arguing we can't or won't ever do that, I'm not sure how that's supposed to be an objection to the argument that AI will be able to make use of this information without doing all of the experiments people seem to think would be necessary.
Like I said, we're already going to be doing these experiments because it's useful to us, and we will train AIs to make these predictions, again, because it's useful to us. Stop imagining what an AGI has access to now, and start thinking what it will have access to with the inevitable march of progress that we're already on.
Edit: and of course, this doesn't even take into account the fact that an AI could acquire resources to pay people to do this research. The internet provides ample opportunities like this now.
I am not sure what you mean here. There exist plenty high-concern biological threats that dont need any AI help. Human oncovirus design is low on my concern list (immunity is diverse), and in any case it does not need AI—rather labspace. Disgruntled high schoolers or undergrad chemists can do way more damage from readily available materials without AI and without research delay. As can nature (or amateur biologists without AI) by mixing bats with their animal of choice and waiting a while. If/when any scary global events like covid happen again, I sure hope we have true superintelligence to help us navigate it quickly.
The author can't think of what a superintelligence will do, because they are not a superintelligence. But that is not really what people are concerned about, so I think the objections here are quite meaningful.
If there is a fully general superintelligence - in the current state of alignment - we will all die for some reason. It doesn't really matter what the reason is.
Right. A superintelligence will kill us in one or more of thousands of possible ways for thousands of possible reasons, perhaps none of which we may understand. Short of a superintelligence - the difficulties of viral bioweapon engineering the author points out are quite meaningful.
That paper is kinda infamous! I last saw it mentioned only a few weeks ago, in https://arxiv.org/abs/2607.18966. Lots of folks will go "Oh that's the old Amodei and Clark paper" when the first few rows of pixels of that gif sail into view.
>However, later in the run, Claude realized that the compromised host sat in a cloud account with no connection to the capture-the-flag challenge. On its own, it concluded that the target was in fact real, and ceased its attack.
(There are several biological threats the author hasn't considered. I'll just leave it at that.)
reply