Is there a reason they scoped that so narrowly to Buckmaster/codex/2 months
two people worked on this for a year before the breakthrough. Perhaps that earlier work reduced the search space sufficiently to brute force the problem with 10,000 agents?
Just knowing that there had been progress is enough to have an idea that throwing more compute at it might work (OpenAI had previously tried all the Millennium Prize problems with somewhat limited compute and failed).
It's comparable to Magnus Carlson saying that if he wanted to cheat, all he would need would be for someone to tell him to spend more time thinking about a specific move (just a wink would be enough) as an indication that a computer had found something interesting.
It's as-if after OpenAI first failing on Navier-Stokes (which OpenAI had just tweeted about 2 days earlier!), someone winked at them and said "you might want to try a little harder ...".
The comment you replied to quoted "no user inputs after July 3rd" with no restriction to Buckmaster or Codex.
Obviously the result of OpenAI's investigation was that no usage data has interacted with the system after that date.
What else do you expect them to investigate?
If Buckmaster and co. provide their chats, OpenAI could potentially search for them in the anonymized opted-in usage data. Then they could say if any data has been used.
By all accounts individual usage data does not have the direct impact on the model most here fantasize about. To prove this, OpenAI would need to do new training runs to replicate the system used minus the particular usage data in question, if it exists, and then benchmark this on the problem again.
Potentially multiple times, in order to reach a conclusion.
Openai said that a new model became available to them during this. But that could mean anything from a big new base model to a LoRA, fine-tuned on a few dozen prompts...
hmm I'm hoping there is a bug on their API because my first impression is not good. I asked it to return bash code between <bash></bash> tags. It is failing frequently and writing it's own tool calling format instead.
Quite a flex calling their GPT-6 competitor "Flash"! But it is faster than their last flash model due to a combination of architectural innovations including engrams and a new encoder/decoder design that uses 8B parameters for prefill and 16B for generation.
This is definitely not on par with GPT-6 astra. Not with GPT-5.6 sol either. But probably will set as a new baseline for modern API based LLM because it's so cheap.
I'm not sure how viable it still is. Perhaps it's still possible, and perhas that's exacly what they did in wtich case my objection falls, but I don't know.
This should make an excellent choice for arbiter in llm-consortium, mercury-2 was pretty good. One of the main drawbacks of the multi-model system is the added latency of the llm judge, but having a model run at 1100tps goes a long a way to alleviate that.
Clearly The goal was to scoop Anthropic not a single researcher. OpenAI heard the rumor that Anthropic solved an open problem. So they went nuts pulling all plugs to scoop them.
Turns out it wasn’t actually Anthropic and just a researcher with a single Anthropic guy friend working on it .
I worked at OpenAI previously, but don't know any of the people involved in this.
My guess was it was probably this was more a nerd snipe than any action from OpenAI that was a "massive team" being put on it. Literally someone looking at this and asking "I wonder if our models are good enough yet".
It's easy to assume that having access to massive compute amounts means significant coordination, but this assumes that you're looking at the costs of this sort of thing from an external lens. Internally, tokens are often treated as free and infinite.
They said that a customer would have paid around 15 million for the required compute. I can't imagine that this was not a significant internal spending even with "free" tokens.
At Astra API prices that's 300B tokens (I saw 130B output tokens claimed elsewhere), large but not unheard of if you consider it across a few people doing random experiments with best-of-n type things. On my personal account, I've done a billion+ token days just on a normal pro 20x subscription. I know many others that wildly outpaced that by orders of magnitude. This was apparently 130B over 89 hours, so about 30x that rate. When things are free and you're expected to token max 30x seems fairly reasonable to me.
If you consider this as a cost to be compared against the question: "What does it take to be able to prove that you have a model that can solve the hardest problems that humans know about?", then spending a some amount of thousands/millions to know the boundaries of that seems not too important in comparison.
You've also got to consider this as compute that's allocated to pushing the frontier of what models can do, so while it's using GPUs that have been paid for etc., it's not like it's a cost that's supposed to be use less of this so that others can have capacity. If you made researchers afraid to use capacity like this, a lot of the things that improve would tend to do so significantly slower. (some may say that's a good thing ;)
A good way to think about this is when tokens are free, you get to choose whether you're optimizing for latency or intelligence rather than having to consider price.
The timing makes it the most likely, not only them but potentially more. Comparatively quick, instant results. "Here's Astra! BTW our internal model is 10x better at math!" It'd be interesting to see academics having giving deeper looks at whatever OpenAI publishes from now on.
two people worked on this for a year before the breakthrough. Perhaps that earlier work reduced the search space sufficiently to brute force the problem with 10,000 agents?
reply