> Everything you say can and will be trained against you
So, experts are incentivized to seed LLM data with false-leads to confound it. Already, garbage is being published on arxiv and elsewhere, and many sloppy code-repos too hastening the process. Expert inputs will be in more demand to un-shittify.
This is the same problem for software engineers too. I am now asked: what can you do that AI cant ? The answer to this could be intangibles like taste, aesthetics, and insights which collectively fall under creativity, and often accompanies experience. And there are no shortcuts to accumulate experience and perversely the more AI is used the harder it becomes. Soon, there will be a closure of all AI generated solutions, ie all low-hanging fruits are taken. Then, experts will again become needed to guide beyond the AI knowledge closure.
> has lamented this practice as being unhelpful for mathematics
A related point is that the actual solution approach is never revealed. What was the role of humans guiding the agents ? was it fully autonomous ? etc. It is in the incentive of the AI labs to trump the powers of the LLM, but in practice it is humans guiding the agents on the overall approach, This is never admitted. For example, in the announcement on NS there was only an output artifact given but no indication of how it was arrived at, and not even a writeup. This is what disappointed many folks as it was done purely for one-upmanship. As other have noted, the benefit is in the journey or process and not in arriving magically at a destination.
I don't think it's as bad as that sounds; in math people work all the time with conjectures they aren't sure if true, and work out a lot of other interesting math based on whether it is or not. Something like Turing's Oracle machine gives lots of interesting math just assuming one could exist, even if it couldn't. It may be that there are things proved we can never come to a human understanding of, but still keep getting interesting math that relies on it that has aspects we can appreciate and enrich our knowledge from.
there are also attempts to crowdsource human research directions - like the caltech mathathon challenge : https://mathathonchallenge.com these would help models on the same problems at the expense of the researchers. basically, math researchers are the reverse centaurs but they dont realize it.
There is a very active open letter of over 1000 signatures from mathematicians in protest of this event. This event is targeting undergraduates. It previously suggested that math researchers already have no place in mathematics, and presents a limited and heavily distorted view of what mathematics research is.
I'm an AI skeptic, but I don't see how this squares with what the organisers of the event actually say. "It previously suggested that math researchers already have no place in mathematics"? I don't see this.
The website previously said, “What is the role of a mathematician when AI can solve conjectures faster?” but they have removed it, possibly as a result of the letter since it happened after.
Why not 10% chance that it will create enormous prosperity for all ? This is why the average person is increasing pissed at AI in general. That it gets associated with negativity.
I'm old enough to remember when people dismissed all the doom coming from these companies as "marketing". (A thing many of them have been entirely consistent about since GPT-2, or indeed earlier given the founding documents).
I know a few people around these circles; People like this are quite sincere about the risk, and that they think poorly of their bosses and how risk is being handled.
Anthropic’s goal is not anything to do with AI, it is purely to generate more profit than any other company.
Securing/lobbying to regulate their competitors through doom talk, or promising that they will bankrupt every other company because AI will “do it all, better than everyone else” is purely for investment purpose.
As long as their incentive is monetary, it’s not unreasonable to believe this is just marketing.
Normally, businesses spend a lot of money saying "no, our products are 100% safe" and get very upset with people who say they are not.
You are literally looking at someone who resigned because the company is acting unsafely, where other people from the same company are responding to this by saying "we agree, this is all very unsafe", in a community where all the various competitors have signed public statements saying "this is unsafe", where people who have Nobel prizes are saying "this is unsafe", where the philosophical centre of mass in the current research community circles around MIRI and LessWrong where the founders made the book "if anyone builds it, everyone dies" because they are so very tired of having to explain that they mean literally, where one of the major independent research organisations is "Model Evaluation and Threat Research", and where the major model makers have been sued for harm already caused by their systems and in one case have been ordered by the US government to prevent people accessing their model because they thought it was too dangerous.
A normal business would be shut down by almost any of these things independently, let alone all of them together. This remains true regardless of whether or not the leaders are psychopathic enough to think this is a good marketing campaign or not.
If this doesn't convince you there's a risk, what on earth would convincing you even look like?
Like, if people were genuinely doing a thing that had such a large probability of killing all humans... they should all be in prison. Heck, vigilanteism would start to look compelling. I do not understand how somebody can really think "well this is likely to doom all of us so we need to get there as fast as possible because we are, without evidence, the most capable people of controlling this thing."
John Naur said the same of coding eons ago [1]. That code is only as healthy as the human mental model of its maintainers. Code is not the artifact, it is the intuitions and insight and mental models of its human maintainers. For a project to sustain, it is vital that the human mental models are kept alive.
More doomerism. Try to implement a deterministic workflow using agents with the latest models and no humans-in-the-loop, and you will realize what they are really capable of. There is too much unnecessary fear-mongering. All of this is only coming from the 2 AI labs trying to IPO. Not from anyone else.
Exactly correct, they are only capable of tasks that any school child could do; like solving millenium prize problems, hacking into tech companies, or tuning particle colliders. Nothing to see here.
> Truly idiotic moments. Peak of Western civilization point.
Could you please stop posting unsubstantive comments and flamebait? This is particularly bad. It's not what this site is for, and destroys what it is for.
You've been here a long time, but you've unfortunately been doing this so much in recent years that I've begun to wince when I see your username. Not a good sign.
It is now clear to me why the AI labs are sponsoring these mathathons: https://mathathonchallenge.com/. They are basically crowdsourcing human researcher data to get access to promising directions possibly later to scoop others.
This part is lost on many. The value of data is in the theories it confirms or more importantly disconfirms, and defining the trading edge is not easy after accounting for costs. I am wary of black-boxes that produce an edge - not only because I dont know how it works, but also because regimes shift unpredictably, and what works today may stop working tomorrow. That being said, AI can be useful to help automate many routine processes just like any other software.
So, experts are incentivized to seed LLM data with false-leads to confound it. Already, garbage is being published on arxiv and elsewhere, and many sloppy code-repos too hastening the process. Expert inputs will be in more demand to un-shittify.
reply