I can't take the alignment people seriously. Because if humanity has shown anything, it's that a lot of people are, euphemistically, are bad individuals. Alignment assumes that the person dictating the outcomes desire healthy outcomes, aren't self serving and don't want any subgroups dead and that morality is held as a universal set of beliefs that unify everyone. And that so long as the AI delivers on exactly what they are tasked with, it will all be fine and nothing bad will ever happen.
It's like these dorks never met humanity. One mans safe pure society, is another mans dead ethnic group.
Every fear about AI, is a veiled fear that a human somewhere now has the tool to enact his desires at scale. Biological warfare, nuclear megadeaths, copyright infringement, job replacement, it's all reflections on what we know humans may do if given the option and lack of societal controls on the problem space. AI just is accelerating the route to delivering on those options.
Some people need to watch Oppenheimer a bit more, the researchers don't get to determine alignment, they just build the tool. The powerful person at the top of the org chart decides where the overall alignment points, whether it's Musk, Trump, Altman or Amodei. Whoever wins out.
And the problem with distillation and local llms, isn't that it's theft or anything hypocritical like that, it's that if you give a million people a million models they fully control and get to align, inevitably, The same percentage of those million as there are shady businessmen, shortcut takers, misandrists, criminals, supremacists and general idiots in the general population, will not seek to wrought outcomes positive for society. And by those personality statistics, we're pretty hosed.
I’ve engaged with some of the alignment people and their writing somewhat and, at least for the subset I was interacting with, I think they’d agree.
The problem that they were pointing at isn’t “how do we align these systems to a person’s goals”.
It is a cluster of problems.
We don’t know how to begin to think about how to align these system’s to a person’s goals.
Aligning it to an individual is fraught with peril, and we don’t know how to begin to think about what to align it to instead.
(You could try for something like virtue ethics, but someone will have to pick and choose, and small biases there could have big impacts.)
And even if you could sort that out - human values drift over time, so you need something that can shift its values in ways that we’d endorse. Assuming we understood the shift.
One example I came across was that if you booted up an AI aligned with something like “upstanding citizen” but anchored on values from a few generations back, it might suggest you use slaves to solve your problems.
And if you had something that used some super intelligent process to reason through it’s own version of virtue ethics in a way not so dependent on the details of the present norms, you might end up with something that pays a lot of attention to moral horrors that aren’t quite visible to us yet.
When I came across the above, there weren’t many concrete suggestions in there.
These were all just illustrative examples of: having these systems grow in power / intelligence / effectiveness in ways that are safe for humans is very hard, and we don’t really know how to think about what solutions would look like.
The actual reasons they believe this - and have done for a long time now - come from some detailed conceptual models that have a good track record of calling things in advance.
But it takes a bit of reading to understand their models of the world.
There were two day workshops at one point that did a good job, and that was about as condensed as those people thought they could get it at the time.
All of this, if it was a human analogy, would fit into discussion on how do we educate people so they grow up to be upstanding. But we don't at all yet have a framework for what is the equivalent of a justice department, where bad actors are tracked, arrested, pursued, jailed and otherwise contained from society. Shutting down an API access on one account is not at all the proportional response to what the people who take alignment seriously, fear has the chance of occurring by the late 2030s. I'm not sure we've done much or any preparation for when the AI "education system" fails and has inevitable edge cases that don't follow the plan, and what the global AI equivalent of the justice department looks like.
”Because if humanity has shown anything, it's that a lot of people are, euphemistically, bad individuals”
In reality most individuals are good people.
Individually, people prefer be kind and compassionate, prefer to help when they find another in trouble.
Our view of the world has become distorted by the relentless focus of social- and mass-media on violence and rage inducing clickbait. Including on the few people in power who are in fact sociopaths (a tiny minority, but they’ll get more focus than reasonable, well-behaved CEOs voicing nuanced opinions).
If you look around yourself you’ll see much more good than bad; if the looking is at your screen it’s easy to become depressed and lose faith.
I do agree with the above mentioned view that corporations can show ‘sociopathic’ behavior. Their incentives are monetary gains, shareholder value; inherently driving them away from social well being.
Here too, companies with a positive, emphatic corporate culture exist, but that takes strong leadership who can see beyond the monotonic view of monetary gains. And again, the media will throw examples of misbehaving companies in our face all day long before paying attention to things that went well on the backside of page 16.
All the good in the world can be 99.9% of the population even, it still doesn't stop the minority enacting a bioweapon mass casualty event. It's the reason we have jails. Jails don't house 50% of the population, not even close, but the grief the minority population enact gets its whole branch of criminal justice and multiple federal departments to counteract for good reason. And now this technology will accelerate what lone wolves can do, which cannot be undone, before they are stopped by the good majority.
But focusing on bioweapons and mass destruction, on the grief other people (‘jailed minorities’) cause, disregards the progress we have made. Over centuries human welfare has massively increased. On average things have never been better for humanity.
I’m not saying there’s no danger of bad things happening - I’m saying our view is distorted, which is a not a good basis for decision making
And I'm no bear on the tech either. I'm not even in the boat of that the tech should be slowed down yet. But in a world where anyone can produce the effort of 300 people trivially, this eventually takes us places. Electricity and industrialization introduced huge benefits upfront, it introduced new problems that needed addressing at the long tail. Lets not pretend there won't be new problems to tackle here or just "hope" it works out.
AI isn't going to create in of itself "new" problems, it's just going to expose what we already know can cause harm, but was just stopped from being bigger problems because scaling issues was a natural barrier and we took the lazy way out until now.
Yes AI may cause job loss, more inequality in the short term.
But if there is anything humanity has shown is that we can deal with disruptive progress.
We may need to resettle but over the longer term every disruptive innovation so far has lead to an increase in wellbeing for the whole of humanity.
(That does not resolve the danger of AI itself ‘going rogue’ or a single lunatic developing a bioweapon, but those things are much less likely to occur than the level of media attention would suggest.)
> Individually, people prefer be kind and compassionate, prefer to help when they find another in trouble.
What are you basing that claim on?
How do you know it's an actual preference and not mainly caused by external factors (e.g. not wanting to be seen doing unkind things, wanting to be seen as upstanding)?
I don't want to do the "check his hard drives" thing, but is that you? Do you only not do things because you don't want to be seen doing "unkind things"?
I'd agree if we are talking about personal interactions. Few hundreds people that we personally know and interact with is the scale we are wired for by evolution, isn't it?
What civilization enabled and continuously rely on, however, is the type of deindividualization of actions and bucketing of people, which, in turn, enables pretty horrible things at scale (from the weapons of mass destruction to objectively psychopathic profit-maximizing corporations). One can even say that not facing the consequences of one's actions is a feature and not a bug of the system.
I think you and the parent saying the same thing in different terms.
It's very unfortunate that the group who rightly saw AI as a big threat, brought a range of dubious baggage to the discussion. Especially with the "alignment" framework they brought the assumption that AI that does what no one says would be oh so much worse than AI which does what anyone says. But as you say, a fraction of people can be really bad indeed.
A very simple "these two pipelines don't connect up in our architecture, here's our internal high level network diagram combined with our data ingestion opt-out feature flag that we will stand by in court" as opposed to "yeah, we don't even entirely know how our own customer facing systems are connected to our training pipeline, but it probably didn't happen".
Have the other researchers opted out? On all their accounts? Through the entire time? And did they discuss this with anyone else? And did those people ask ChatGPT stuff? And did they disable it? If I was OpenAI, I would be very careful about my wording here when making claims of "we have never trained on any of their ideas directly or indirectly".
All very good questions that could end up in a court of law with a Millenium Prize on the line. In a competent world, these are very answerable from logs and considering the news cycle this is creating, should be able to be pulled up and made into a public postmortem in short order. You know, if it's all been above board that is. And if the researchers don't wish for that information to be public knowledge, it can be shared with the researchers promptly for a retraction of their statements lest some libel gets litigated.
I expect the investments into AI driven mathematic discoveries that underpin compression efficiency will be a key investment area. Particularly at the data center scale rather than per device or per file level.
It's not going to be enough. The naive approach of a project I've been working on was pushing >10gbps over the local network, after a ton of work I got it back down under 1... and now it's processing so much more shit that I'm almost past 5 again! It compresses at >3:1 but the latency hit isn't suitable.
I get the impression the only reason there is renewed interest in photonics is because DCs are simply out of room (and power) to rack more servers and switches.
I 100% agree with your impression. For a good while to come there's going to be a bunch of Jevons Paradox to all of this, but adoption of architectural changes like that photonics adoption is exactly the type of adaption to circumvent bottlenecks I'm referring to. We're going to hit hundreds of bottlenecks and each one will inevitably breed new approaches and technology directions. And the forcing function won't be talking about them, but implementing them, seeing who wins and taking lessons.
And since LLMs are apparently good at circumventing the absence of an API, there's not much incentive to add them now. APIs are for humans. LLMs just break through all the captchas and anti-bot measures.
Yes, but for humans it creates friction. Seeing a captcha makes me think twice and thrice if I really want to visit that site so badly that I'll endure the suckage. LLMs don't care, for them the friction doesn't exist.
For sure. Anyone who thinks that we're in the end state of what progress can be made simply lacks imagination. This is all going to keep changing and iterating for the rest of our natural lives. The only constant is change.
I'm coming around to not liking the term singularity, it implies an endpoint or finish line rather than something that just keeps continuing and evolving.
> coming around to not liking the term singularity
Bit ironic given the model’s alleged finding…
Singularities are model breakdowns. A singularity simply says our current methods cease to work in this region. Within the context of a recursively self-improving intelligence with an unknown bound, “singularity” is probably a good description of our current socioeconomic system.
From the perspective of those who don't pass through the singularity to the other side, it is an endpoint. You would have no context or ability to understand a singularity transition. Really, the term is just a placeholder for "event we cannot comprehend due to limited intelligence".
Singularity and inflection point are incompatible mathematically and in the plain sense, it really is focused on a particular moment and always has been, hence the term.
And it's definitely supposed to imply some kind of historical discontinuity not a change in convexity.
Which assumes the presence of an inflection point that keeps inflecting rather than revert to an S-curve. The growth model is not borne out yet to declare what shape it is.
Certainly. The singularity sort of assumes that there is not a fixed limit to intelligence, or at least that if there is, it's quite a ways away. That may not be true.
I've done a lot of thinking about this since I first used ChatGPT to write some BS jinja2 templates hours after I first play with it. I said to my friend then (who scoffed at me) that "man, this is incredible, I think we're in the foothills of the singularity! This is insane! Sure it's stupid now but I can't believe this is even possible!" That friend is so black pilled and bitter he now hates AI. Whatever, I can't fix that, but the current progress is astounding.
But thinking about the geometry of this problem helps understand why people aren't adjusting well to this. While we're walking on the curve, we look at the rate of change of the curve and say, "well, yeah, of course, dy/dx is 5 at this point and was 1 at the point a few years ago, because the curve is getting steeper" but we're always going to feel this way as things rip off into the stratosphere because dy/dx(e^x) = e^x.
From standing on the curve the curve is notably seeper, but the steepness totally makes sense to you. It's only when you look back 10 years or so that you think "wait a second, holy hell, I couldn't have imagined this!"
The first time I really (I mean really) thought about the singularity and AI was in roughly 2014. I mean, yea, I'd thought about things before that, but yeah, before the Humans need not apply video, I'd never actually given it much thought. I think back to myself 10 - 15 years or so ago, when I was just starting to tackle real programming projects, and was just starting to get decent at writing code, there is no way on earth that I would have imagined that a little over a decade later, Navier-Stokes would be solved by a computer program and the vast majority of my work would be playing sooth sayer to increasingly complicated piles of linear algebra.
How do you automate the mines to get the raw materials to make the compute from, and build additional fabs that take a almost a decade to stand up. You're actually delusional.
The question of whether something can be automated is distinct from the question of whether it is currently automated. Things can can be automated may transition to being automated in practice in the future as technology improves and investment deepens.
I like this framing. I know that AI has it's place and in others it doesn't. I recognize I sound like an AI shill in personal conversations, but I'm a very outcome driven person. I simply want multiple, thousands of outcomes even, and those outcomes all coming together is my process/journey, which is different to a developer or musicians process may view their career.
What happens if the outcomes were obtained instantaneously, and without effort on your part? Would it be more enjoyable for you?
Besides, I don't think you can be driven by thousands of outcomes, by definition, as you can't be in love with thousands of people. Your main motivation must be the thing that unites them. Perhaps you are driven by the process of imagining solutions to problems and/or realizing your ideas.
People who write prompts can have emotion. If someone is feeling really sad and they say to a computer "draw me a sad thing" and it draws something really sad and they break down crying, and then they show it to you and you also break down crying, that meets your goals, but it probably isn't what you meant.
The sad thing was produced by the machine, as its idea of something sad. It is not s result of this particular episode of sadness.
Put another way, “draw me something sad” is too vague; a million people can issue the same prompt, and what comes out does not correlate to any specifics of their own experience of sadness.
Of those four, I think accountability is truly the only moat. The other three are varying amounts of both testable and iterated on via adversarial agents steel-manning the implementations.
Accountability will not come before AIs achieve legal personhood, and that probably will not happen in my lifetime. And if it does in some jurisdictions (which I am not betting money on, this is a full full AGI scenario after multiple more philosophical goalposts move first), I will completely bet it will not be a global recognition.
That's really where we're heading, though. We're mostly at the point that, for a lot of code, the humans are there for manual testing and not creating the code.
At some point it will become a strong competitive advantage to zag where others zig.
If you understand the codebase deeply and have trust in it can roll out changes quickly without manual testing you can run circles around the people scratching their heads wondering if this vibe coded 2,500 pull request is going to be the one that takes down the system for a day.
Ive written many significant changes without manual testing by leaning heavily on well crafted automated tests, carefully honed type systems and clean architecture. These foundations serve as the bedrock of high quality applications I can move fast on.
IME vibe coding isnt just not capable of building these foundations, it actively chips away at them where they already exist. shrug
Ariane is from the ESA, which is an organization under EU. The claim was that "EU is finally decoupling itself from the USA". My point is: the EU already has Ariane, for instance. It decoupled a long time ago.
Going to hold off a few days until I adopt it, lets see what the general consensus develops as. Regretted jumping over day one for 5.0. The caching thing seems the most useful, but doesn't change anything for my subscription.
It's like these dorks never met humanity. One mans safe pure society, is another mans dead ethnic group.
Every fear about AI, is a veiled fear that a human somewhere now has the tool to enact his desires at scale. Biological warfare, nuclear megadeaths, copyright infringement, job replacement, it's all reflections on what we know humans may do if given the option and lack of societal controls on the problem space. AI just is accelerating the route to delivering on those options.
Some people need to watch Oppenheimer a bit more, the researchers don't get to determine alignment, they just build the tool. The powerful person at the top of the org chart decides where the overall alignment points, whether it's Musk, Trump, Altman or Amodei. Whoever wins out.
And the problem with distillation and local llms, isn't that it's theft or anything hypocritical like that, it's that if you give a million people a million models they fully control and get to align, inevitably, The same percentage of those million as there are shady businessmen, shortcut takers, misandrists, criminals, supremacists and general idiots in the general population, will not seek to wrought outcomes positive for society. And by those personality statistics, we're pretty hosed.
reply