To be fair, the post does start with "This is a guest post by Timothy Gowers", feels pretty clear already. But of course, assumes reading beyond the title.
I answered this to myself - stop worrying about LLMs. It's pretty simple: due to Curry-Howard isomorphism, programming languages are just notations for some type of formal logic.
Now ask yourself a question, what language do you want to maintain the programs in? Do you think natural language is going to be easier and more maintainable than formal logic?
The answer is no. So you need programmers, people who can read the formal description and adapt it to new requirements.
LLMs are amazing technology, but the truth is - natural language just kinda sucks. Therefore, you don't really need them (see also https://en.wikipedia.org/wiki/AI_effect ).
I think people love LLMs for the same reasons they love magicians. But just like the magician employs a hidden trick, LLM just runs some algorithm you don't see or understand.
So worrying about LLMs taking programming job is kinda like worrying that a magician will take a warehouse worker job, because they can levitate stuff. Meanwhile, we already have automated programmer - it's called a compiler.
Natural language is valuable for the things it doesn't say. The ambiguity is core to the functionality. Which can be very helpful when navigating social complexities.
And then written language is also valuable for the things IT doesn't say. Under the theory that 90% of communication is non-verbal, then writing lets you say things without having to communicate that other 90%. Which can be very helpful when negotiating something, for example.
I would agree with EDW, and I have argued here in a similar way.
I am not against use of NL in negotiation or poetry. If you find ambiguity useful there, be my guest. But engineering specifications, mathematics, as well as other sciences or even philosophy would IMHO benefit from more rigor.
I also strongly disagree with the notion that logical or programming languages cannot express ambiguity. (It actually took me many years to understand.) I used to think you need something like fuzzy logic or probability, but that's unsatisfactory in some ways. Eventually, I settled for a really simple understanding of the problem.
Take lambda calculus for instance. I define the term to be ambiguous iff it has a normal form. So it is ambiguous if it expects additional argument, which resolves (part of or all) the ambiguity. Terms with no normal form are completely unambiguous, their "output" is completely given.
In classical logic, this corresponds to formulas that are conditioned on additional assumption. Again, the extra assumption can resolve the ambiguity.
So it is kind of my conviction (although we could show that by translating an LLM as a program into LC) that all the words in natural language can be formalized as sufficiently complicated lambda terms, that all have normal forms and react to each other in a way that resolves some ambiguity without ever resolving all of it.
LLMs do not execute natural language. Natural language describes a problem or request, and LLMs generate and test formal logic they predict will satisfy the request.
I disagree with each sentence for a different reason.
LLMs interpret (so, "execute" in a way) natural language in the sense they have internal logic that assigns to the sequence of tokens in context a next token. If we delineate the input and output into a series of logical statements, we can think of it as a program that builds a logical statement from a list of input statements. So it encodes derivation in some logical system.
However, the internal logical system is informal in the sense that the above rules are not guaranteed to be sound on the fragment of classical logic encoded in the natural language. It is a close approximation, though, so it often works.
To add, half of my problem with natural language would be resolved by agreeing on exact definitions, which is kinda what LLMs do internally. However, they don't surface this formalization very well(even with open weights it's difficult), which makes it pretty unusable.
> LLMs interpret (so, "execute" in a way) natural language
You could say the same thing about human programmers, but I've never heard anyone say they think that programmers "execute" Jira tickets.
> they have internal logic that assigns to the sequence of tokens in context a next token.
I don't think this means what you think it means, because it has almost no information content relevant to what we're discussing. The probabilities that are most relevant at the level we're discussing are satisfying a reward function from post-training, which approximates to:
"What is the likelihood the solution the agent is pursuing will be marked correct by the automated grader based on the full prompt and other context provided?"
It still has to predict the next token but that isn't based on a likelihood of that token appearing in a corpus of internet text consumed in pretraining. That was eons ago. Every predicted token is shaped by the probabilities of the predicted solution, which must already be very specific and shaped completely by the request and associated context that is built during investigation of the same.
"You could say the same thing about human programmers, but I've never heard anyone say they think that programmers "execute" Jira tickets."
Yes I could. We have "executives", for starters. And first "computers" were actual humans.
"That was eons ago."
Yes, technically I should call them LRMs (large reasoning models) not LLMs. But that doesn't seem relevant here, to my point they encode some logic (which we want to be close to classical logic, i.e. behavior of words like "true", "and", "not" and so on matches).
That isn't what executives do, and I'm not talking about semantics. If you believe that the term "token predictor" has any relevance when discussing the capabilities of these models, you are misinformed. That is the inner, inner loop and it is simply the substrate through which reasoning and action is expressed.
When you want to change the resulting program (“formal logic”), you generally don’t change the prompt and regenerate everything (like you would do with source code). Instead, you provide new natural-language instructions to have the LLM perform the change. This means that you don’t have a reproducible specification on the level where you are describing the specification. Therefore the thing that you are maintaining is not a natural-language artifact, it’s effectively still the formal description (program code).
And that’s why the parent comment is saying that we need people that understand the thing that is being maintained, i.e. the program code, which is the source of truth about what is being maintained.
> we need people that understand the thing that is being maintained
We do for a lot of software today, but not all of it. I think in a year we'll need them for less, but I'm not sure how much less.
This claim isn't supported or justified by talking about natural language versus formal language. For the last 50 years or so the people shaping a lot of software in the most important ways are often using only natural language.
The product manager doesn't understand the code today. They write PRDs and Jira tickets and comments in Slack, and software comes out. They have people test the software, report bugs, more software comes out. Eventually they decide its close enough to their vision to ship, without ever understanding or looking at any of the code. They can do that with human programmers or agent programmers. The former holds up better in larger systems, but I don't see any evidence for the proposition that this is due to the limits of natural language.
Developers are making a lot of important decisions in detail that the product manager has little understanding of, and these often lie exactly in the disparity between natural and formal language. What the product manager writes isn’t a full specification of the system.
I totally agree with the skepticism that we'll ever get to a point where natural language becomes the "formalism" and stop needing people who understand the actual formalism underneath. And I agree that you don't need LLM based tools.
But things that you don't need can still be (and often are) incredibly useful. Nobody needs an IDE, nobody needs vim or emacs or bash or even compilers or assemblers.
But we have all those tools and they are useful. That is, their utility is net positive.
Using LLMs to generate code currently also has (wildly) net positive utility. Maybe that will change because some part of the calculation changes. But this is the situation right now.
The bit about "but the UI" sounds like a developer thinking "UI is a solved problem" in the same way MBAs are told "coding is a solved problem".
Product people do seem to think it worth exploring "personalized software for everyone" like literally each person gets their own custom UI. This does sound like a support mess but it's not obviously wrong when you think about the Microsoft Word alternative.
I agree LLMs are useful tools, but I am dismayed by a lot of cargo-culting around them, which happens because we don't understand them. I think we will have much better tools when we understand what is the expectation and what is the algorithm they run.
Somebody else said that the magician analogy was poor. I like magic tricks, but it took many years of cultural change (influenced by people like Houdini, Randi, Penn & Teller) to stop illusionists (and mentalists) make claims they have supernatural abilities, or people believing it on their own (a magician pretending to be able to catch a bullet was shot by an audience member who didn't understand the distinction).
It is detrimental, I think, to treat LLMs as if they have magical abilities ("superintelligence") rather than understanding they just run some clever algorithm. The fear for (programming) jobs comes from that framing; nobody fears of their job because of compilers, since compilers are understood.
(And it actually runs against kind of "socialist" framing of the problem, which I agree with, that is why should people be worried about the jobs in the first place, when society is getting richer as a result of better tools?)
I see what you're saying, but I don't really think understanding how they work is the problem. We do understand how they work. You can go read the many papers that have been published as the technology has been developed. It is not mysterious. We don't need to understand the "clever algorithm", we already understand that. The sense of mystery is because the results of the algorithm are non deterministic, and because the number of parameters that determine the outcome is so large that it appears to exhibit emergent behavior.
I guess I agree with your general premise that tools we use should not be treated as magic. But I don't think this is a problem to be solved, it's possible to learn how these things work, and people should definitely do that.
I disagree we understand LLMs, your view is quite reductionist. It's like saying we understand human brain because we understand physics and evolution, or that we understand some program because it's just a binary code in a known instruction set.
I think Dennett's theory of intentionality (see https://en.wikipedia.org/wiki/Intentional_stance) applies here. We do understand LLMs from physical stance (ML algorithm and inference), but we don't fully understand them from design stance (it's internal workings have been evolved so it's hard to tell the functional units) and from the intentional stance they are a complete mystery.
And I talk about obstacles to this understanding elsewhere in this thread.
I agree with this but what I've been trying to answer the last few months is if there was an optimal language for the spec. As with you, I don't think it's English Markdown, but I don't think it's Java either. I also don't think it's Gherkin, Lisp perhaps? I'm still searching.
Well.. I think this is a big open problem in philosophy.
On one hand, you have things like Lean (calculus of inductive constructions), these are relatively simple formal logics (just in more practical notation) that let you define any conceivable type, which is akin to specification.
On the other hand, there is a rich set of modal and fuzzy logics that can help with aspects of reasoning in natural language. I think these can be defined in the former, but nobody has really made a good agreement as to how.
So the main difficulty is for any such language to gain traction, people who speak it.
Instead, we trained LLMs and they came up with something (evolved to reason). I think the future philosophical research will need to answer what exactly do LLMs bring to the table in terms of formalization of natural language.
Your analogy is poor, and you are missing a very important fact.
Most of the human written code, in places where that code needs to make money, is decidable either entirely or in large parts. I.e without running the code, you can take a domain of inputs and build a complete range of outputs solely by looking at the code.
The way that works in your head is that you are effectively doing a compilation to a logical like structure, which then you can use to infer what the output will be from what the input is, and its a direct mapping that is invertible and separable, so if you know what the output should be, you know what the input is, you can pinpoint the exact location where it breaks. Thats how humans write code.
If thats not clear, imagine a piece of code that splits strings by spaces, deletes the empty strings, and returns the number of words in a string. The fact that you can say that if you want 3 words, there should be maximum 2 sequences of continous spaces between words, is you effectively transpiling that program into a latent space inside your brain neurons and inverting it.
LLMs essentially do this, with the added advantage of having been trained on a HUGE number of codebases, so they can recognize patterns that a human cant.
Where LLMs struggle is complex behavior - they can't simulate things like a human can and choose the best course of action. Even harnesses for agentic loops that can auto run and debug code can't match what a human can do in this regard (hence why self driving still sucks rn).
So moving forward, being a good coder isn't going to be about writing code, or even about prompting LLMs. Its going to be all about whether or not you can design good custom agentic loops, which necessarily involves knowledge of the model at hand (i.e what words you have to use to get it to do the right thing). This will be especially true as investment into "private" inference grows where companies will be using smaller models that have less detailed RL and thus will need much more guidance to do the right thing.
I disagree, to keep it short, what you're describing is not understanding, it's superstition. And I think it's a wrong direction of engineering, relying on some sort of irreproducible expert intuition, one that has been successfully replaced by enlightenment and scientific method.
There are 3 major obstacles in understanding LLMs:
1. They use inscrutable internal language of embeddings
2. They communicate in natural language which is itself ambiguous
3. The weights and training inputs are being hidden as a "trade secret"
"with the added advantage of having been trained on a HUGE number of codebases"
This doesn't really mean much unless we understand what is the quality and relevance of these sources for the problem at hand. Without this understanding it's just a superstition.
>They use inscrutable internal language of embeddings
No different than the electrical signals in the intermediate neurons in your brain that comprises the latent space where all the processing happens
>They communicate in natural language which is itself ambiguous
They lack one-shot precision, sure, but it doesn't matter. They are precise enough with refinement over multiple prompts.
>the weights and training inputs are being hidden as a "trade secret"
For frontier models that make the company money through api pricing, sure. There are plenty of open source models that can be used for the same tasks, which have open weights.
>This doesn't really mean much unless we understand what is the quality and relevance of these sources for the problem at hand.
All of the modern models are RL trained on specific tasks when it comes to coding. I.e the initial training run learns to predict the next token based on context, from all the available texts, but then the RL runs specifically train the model in a harness where it produces code and RLed to produce correct code with specific formatting.
Not really sure if it's sarcasm, but let me make a side remark.
It's really stunning how much more effective the "Standard ML" notation (embraced by Haskell, Lean etc.) is compared to writing proofs in classical logic.
This "UX problem" is, I think, the reason why is mathematical community embracing automated provers maybe 50 years later than they could have. Automated people wanted the better language, but the mathematicians largely resisted.
So seeing this, it would be preposterous for me to think that any language, natural or not, has the last say in this. We're gonna be stuck with learning new languages and formalisms for a long time.
I am not sure what your counterargument is. But in mathematics and computation, people have tried for at least 150 years to move away from natural language, and figure out stable foundations that can be externalized. I think there is a good reason for that - you save time correcting errors due to different interpretation.
Well, almost all scientific domains have developed a form of structured and formal language, because natural language is too ambiguous. There's "code" everywhere, not just in programming.
We are in fact finding out precisely why natural language sucks in real time, as we have all kinds of catastrophic errors with people who think this is finally the time for complex language to prevail over pesky nerd language. The only difference is that more people seem to prescribe to the "you're holding it wrong" handwave when said catastrophes are pointed out.
I can appreciate your original point, but this degenerated into naivete. The is-ought problem exists in (and was formulated for) natural language, and ambiguous formal languages are childsplay. There are further problems (ignorance of the consequences of reality being finite, misunderstanding the operative layers of interpretation) but these two alone are disastrous by themselves. Don't mix up convention with implicit substance. There's a very basic philosophical lesson you're missing, and I'll let you in on the secret: The labels aren't actually descriptions of any property. The distinction is indoor baseball. Notations, syntax, semantics. What you want is signal, and you can transmit that any way you want. Writing and drawing were once the same thing. Still are.
You are right, and that is precisely the issue. When complex logic fails, we don't usually blame the machine. When natural language fails, we as of late seem to be trying to anthropomorphize a machine that cannot be held accountable. As if understanding natural language suddenly means it understands morals.
No offense but you're just making statements without backing it up with anything. "Clearly superior", "finding out that is true"..
How are you going to prove what you said? Natural language is not enough for that purpose. You need formal logic, quantifications, specifications, the foundation of programming. Superior to what, and according to what metrics? What truth in computation are you talking about, and how can we know and confirm it? Not with natural language, but with numbers, mathematics, the building blocks of logic.
It's basically a convention. The alternative is to raise interrupts of course, but that might be application specific, or use other invalid instructions than the designated one, but they might work differently on other processor types.
There's a bit of convention and practicality The only thing you really need is that your "fatal error" instruction and "syscall" instruction can be reasonably discriminated without needing to set registers at the call site. Needing to set register to identify a fatal error is not great for code size, especially in languages that generate a lot of them (memory safe languages, mostly).
Though, yes, convention does play a role. On ARMv8 you get both SVC <imm> and BRK <imm>. SVC and BRK raise different exception codes (which satisfies the "easy to distinguish requirement) but in principle you could just use BRK with a well-known immediate and eliminate the need for SVC since BRK's immediate is reported in the exception status register. And, anyways, if you have an SVC instruction and a BRK instruction, you may as well use the SVC instruction for syscalls since it's right there.
ARMv8 also gives you a a UDF imm, for a guaranteed undefined insn with an immediate payload.
The reason to want a true UDF imm with an immediate comes down to it being pretty solidly guaranteed that it's going to turn into your language/OS equivalent of a SIGILL insn. In theory an OS could by convention allocate some subset of BRK space for arbitrary userspace purposes, but in practice none did, so trying to use BRK gets you dumped into a debugger, or doesn't have consistent behaviour. It's nice for userspace to have something that doesn't need active OS support.
Graeber actually points out that centrally-planned economies had formally higher work hours than was the reality (people slacked off more), and that it was akin to workers collectively making a choice working less for the overall lower production.
But the obvious problem is that it's still a societal taboo, so it's not UBI because it's not universal. It's probably lot more efficient to actually openly embrace shorter work hours, so that people can stop pretending and choose to do something more useful.
Also, existence of BS jobs is often driven by high status individuals, so it is comparatively less free than UBI.
I was bound to agree with you before I RTFA. The classification is useful for a psychiatrist, i.e. someone who wants to help the person.
"Bad coworker" can have many causes. In prolific AI psychosis, I think the assumption (differential diagnosis) is, they became a "bad coworker" due to use of AI, so they were a "good coworker" before.
No, the term itself is not stigmatizing. People stigmatize legitimate psychiatric diagnoses, and you're perpetuating this idea in your comment, by blaming the term for the moral failure. These labels don't exists to stigmatize people, but to study the problem and help people.
> Maybe you can upgrade digital systems until they are 'fully fortified' in theory, but humans have an upper limit what they can understand / perceive / critically analyse (not to even speak about engineered high pressure situations) that can't be upgraded.
There is an answer to that - simplify and abstract. Lots of human software is unnecessarily complex, often caused by backwards compatibility and general human creativity.
Take sendmail vs postfix as an example of this process.
Not to mention, the idea that Germany, 3rd or 4th largest economy on Earth, cannot defend itself on its own and needs American troops "to defend itself" is just ridiculous.
The Americans are there as a front of US empire (IIRC Rammstein is being used for Iran war).
reply