On the SDK front, you tried just having an agent reimplement the model you're interested in and just use the weights? I've taken to treating off the shelf implementations as reference implementations anyway, because I can often squeeze out significantly better performance for my configuration and use case by having Codex hammer at it for a few hours.
It's been pretty ad hoc, but my prompts are nothing special. Things I generally do:
1. Top end model on high/xhigh thinking (last time I did it it was Sol xhigh I think)
2. Make sure it creates some representative fixtures of different sizes and sets up a good testing, profiling and benchmarking loop that doesn't require my input.
3. Make sure it has access to reference implementation code
Edit: Oh and one obvious pitfall that for some reason I still have to remind even smart models of from time to time: make sure it knows not to try to parallelize its benchmark runs. I've occasionally had an agent struggle to figure out absolutely nonsensical data because it tried to run multiple tests on the same compute hardware simultaneously.
I think it was on qwen 3.5 9B or thereabouts that I saw a ~30% decode rate improvement on my M2 Mac vs MLX. That’s probably the best I’ve gotten.
Keep in mind though that this is also sometimes use case dependent. Off the shelf implementations are generally pretty good overall, but can have pathological behavior on specific workload shapes you care about. So do this as a somewhat later optimization, and particularly when you see performance characteristics that don’t seem to make any sense.
The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.
Note that Vulkan is only somehow better than anything else out of Khronos, because at least for that API someone is paying LunarG for the Vulkan SDK, and NVidia has a vested interest in now pushing slang as the shading language.
The moment LunarG no longer gets sponsored it will be back to "here is a standard, all the best, the community will provide the tools".
On the SYSCL front, that is mostly driven by Intel, and their own OpenAI tooling, there are hardly any offerings for SYCL from anyone else. Thus it is really only pseudo-open.
Consider that for the others you are supposed to build your own SDK from random pieces of libraries, not all of them playing well together.
In that regard, the Vulkan SDK is definitely an improvement.
However using Vulkan on Android is actually hardly any different, given that there is no SDK as such, only the libs and headers get dumped into the NDK, there is a github with samples from Google and that's it.
So bad that they failed to convince anyone to drop OpenGL ES from Java/Kotlin, and use NDK alongside C or C++ for Vulkan, thus now there is WebGPU implementation for Java and Kotlin consumption.
I do agree with the Vulkan code itself, related extension spaghetti, and overall state of drivers, which has become visible enough to have been part of Vulkanised talks for two years in a row now.
This is at the core of the matter for me, and my knowledge is too weak to understand why this is the case. I don't enjoy the idea of relying on Nvidia's stack for GPU compute, but the alternatives I've tried (e.g. Vulkan compute) are higher friction to use. I am trying to reconcile why; Nvidia shouldn't have this moat.
My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.
All commercial APIs have this moat, which is why other than Carmack famously advocating for OpenGL, the large majority of other devs never cared, they implement an API agnostic engine, use the best API for each platform, and move on with the rest of the game.
Even Carmack later admitted that DirectX had become much better than OpenGL, but they weren't going to change given the studio culture.
Vulkan compute is not really designed or intended to be a CUDA competitor, its feature set is much more restricted, and Vulkan host side code is much more verbose than CUDA. OpenCL or SYCL are much closer in features to CUDA. I found that when using SYCL on Nvidia, debugging symbols etc can be passed through and you can use tools like NSight Compute to profile it as if it were CUDA.
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.
From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension.
The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
So when I said "a couple of nvidia engineers", I indeed meant Jeff.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
> I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable
__VK_ENABLE_DEVICE_GROUPS=1.
It could easily be a competitor to Cuda, if it just made things easier. Like, why does it take 50 lines of code to allocate memory in vulkan, and just one single line in cuda? Vulkan should just provide a single-line gpuMalloc convenience function. And not just for allocation, for all the other nonsense as well.
I would too, but sadly that's Khronos' job to organize, and they've had trouble getting American vendors to work together.
It's likely that CUDA will continue dominating until they put aside their differences. The current MLX/MPS/ROCm ecosystems are too fractured to threaten Nvidia.
Yeah talking about the (vendor-preferred) compute part here
Vulkan's SPIR-V dialect is substantially different from the OpenCL one, notably with the former having structured control flow. They're incompatible between each other.
I'm talking about holistic efforts like OpenCL, and standards that would be equivalent to Nvidia's "Compute Capability" versioning.
The basic underlying tech can be agreed on, but Apple/AMD/Intel all have different GPU priorities that limit their ability to agree on a CUDA-adjacent hardware platform.
Ultimately AMD missed the most important boat with their woeful support for GPU compute. 10 years ago they needed to go all in and offer something to compete with CUDA - whether that was internal or something standard from Khronos. They just couldn't commit to it as a business, didn't have the vision.
Intel missed for a different reason - they didn't invest in the product space at all for decades. In terms of units they had the most market share (millions of Intel Integrated Graphics chips), but it wasn't seen as important. Bare minimum to render Windows and Office UI, nothing more.
What do you mean by holistic? SYCL is an open versioned standard that allows for vendor specific extensions. The problem is not that there isn’t a proper standard, the problem is that many hardware vendors - or software developers simply don’t want to adopt it.
Intel (via Codeplay) was handing it out on a silver platter - Nvidia on SYCL, full top chain, and people still wouldn’t want it.
Isn't OneAPI a good example of the problem, alongside Mojo/ONNX/TensorRT? The industry doesn't need a fifteenth competing standard. They need hardware buy-in.
By holistic, I mean hardware architecture cooperation. Nvidia can hold onto their lead forever if GPU designers fight over what a GPGPU hardware baseline looks like. The current ecosystem fragmentation is not competitive, and future fragmentation probably wouldn't work either. I think the fastest way to kill Nvidia would be a hardware consortium.
The problem with OneAPI is naming. It leads people to believe that is another competing standard where in fact is is simply just an implementation of a standard compliant SYCL compiler. If it just had been named “Intel SYCL compiler”, similar to the existing and accepted Intel OpenCL compiler, it would have been easier.
What would you expect the hardware consortium to coordinate on? Unified ISA?
They were maintained by Codeplay - paid for my Intel. Nvidia can make contributions anytime they want, and here is the problem: Nvidia does not want to. Until each vendor starts pitching in with contributing their backend to an open standard, you will have to rely on others doing it for them.
It's impossible to have an open standard. Hardware accelerators are nothing alike and have different perf characteristics. Each kernel is tuned to the hardware. The idea of writing a performant kernel in opencl is a fantasy.
Source: worked at a bunch of accelerator companies in the kernels or equivalent team. They're nothing alike.
C ran on computers with 8/12/16/18/24/32/36/48/60 bit words, computers with 8/9-bit bytes, 6/9 bit characters, computers with word and byte addressing, with one's and two's compliment integers, with flat and non-flat memory models, with different endianness & different alignment rules.
Early graphics APIs ran on GPUs that were just about as different form one another as that.
We could easily figure it out, were it not for the trillion dollars incentives to not figure it out.
For a performant portable language, we’d have to go to a higher level where you describe what to do and leave the how to do to the compiler. It would then need to be able to adjust memory layout, access patterns, data type choice to the underlying hardware. I’m not sure if this is possible to do reliably - the closest we have right now may in fact be highly detailed plain English descriptions of the algorithms fed to an LLM prompted to produce assembly.
It's not possible to do reliably. None of the models in current use today use any esoteric math. It's extremely easy to implement the math behind both the inference and learning of all modern models.
Not everything is AI and dot products of massive vectors, there are still applications that do other maths on GPUs
My thinking was rather that most of our current programming languages put memory layout fully into the programmer’s responsibility - I can think off hand of a language where the compiler makes performance decisions like whether your structure are SoA, AoS or SoAoS, what alignment, padding, strides and float types to use.
Automatic decisions about when to use cooperative loads through shared local mem versus gathers from global mem and hardware caches are also something that such a hypothetical compiler would have to make.
I mean if you don't care about perf, opencl does what you want, and exists today.
As for ai and matrix vector performance... I mean matrices are extremely useful across many domains. The hands off language that exists today is called blas. That's fine but won't lead to the best perf on any GPU today.
SoA and AoS data layouts are not even a worthwhile point of contention. Same with shared v global mem. Today's hardware has specialized memory depending on the operation. The hardware on these processors is so specialized as to make anything but first class support for the feature be completely pointless. If you look at Nvidia code that's open source even they will special case a lot of their chips. Literally if you want the best perf you write the kernel exactly for the exact chip. That's intra vendor .. you can only imagine inter vendor
The (naive or forward thinking depending on your opinion) is that cheaper 'inference' will result in more demand, like the more efficient Watt steam engine increasing consumption of coal.
I predict that within 100 years, data centers will be twice as powerful, ten thousand times larger, and so expensive that only the 5 richest kings of Europe will own them
It reminds me of "640KB of RAM is enough for everyone". Look where we are right now. We always find a way to spend resources. I was frustrated with my last laptop because of frequent swapping/OOM, so I built a desktop with 64GB RAM during covid. I thought I would never have this problem again, I even disabled swap. Now I regret I didn't get 128GB instead to fit bigger local LLMs. But even without LLMs, my Firefox crashed with OOM just yesterday. Apparently I open too many tabs too quickly.
I used to think "wow, ADSL is so fast, I will never need faster internet connection" and now I need fiber to sync 500GB dataset fast enough so I don't get frustrated. Gosh, I remember the feeling when I upgraded from 14.4k to 56k dialup modem.
Energy usage for computing by humanity will only go up with time. It should be obvious that most of our energy will go to computing for anyone who's worked with computers. At some point you exhaust the kinds of things you can do in the physical world, but you can compute infinitely. There's limits to where you can go but not to what you can think about.
Never heard of A/I and I have no idea what they are about, but rather than a government communication this statement reads like an angry rant with lots of unhelpful name-calling.
I would expect a government to be professional and just stick to the (proven) facts and avoiding labels; that would also be more convincing. Oh well.
This is the new Republican mainstream, the same mainstream that brings us AI generated videos of the Obamas as apes. They have no shame and are likely irredeemable (certainly their behavior is inexcusable).
And ultimately these statements aren't for the normal citizens. They're for the truly strange, the racists, and the trolls.
> The anarchist cells responsible for sabotaging rail systems across France, Italy, Germany, and the Netherlands in 2026, relied on A/I Collective tools and services to claim responsibility for the attacks, publish official communiqués, and disseminate manuals for constructing improvised incendiary devices to target railways and other critical infrastructure.
> The anarchist cell responsible for sabotaging the Transalpine Pipeline (TAL) in March 2026, which temporarily halted crude oil flows to Austria, Germany, and the Czech Republic, relied on A/I Collective tools and services to claim responsibility for the attack and publish official communiqués, as well as to disseminate a manifesto calling for additional attacks on critical infrastructure and referring adherents to another A/I Collective platform, which provides detailed sabotage manuals as well as maps of vulnerable critical infrastructure across North America.
> The far-left extremist network responsible for multiple arson attacks and acts of sabotage against Germany’s rail and energy infrastructure between 2011 and 2026, including a January 2026 attack on Berlin’s power grid that cut power to 45,000 households and resulted in a fatality, relied on A/I Collective tools and services to claim responsibility for attacks, publish official communiques, and disseminate a manifesto calling for additional attacks on energy infrastructure worldwide.
Is the car manufacturer responsible for their use of a car? The power tools manufacturer? The paper sheet manufacturer? The restaurant that fed them?
Technical intermediaries could be held responsible specifically for hosting the manifestos (under freedom of the press laws), but that requires an actual due process. Meanwhile, 100% of the authors and websites who inspired or relayed Anders Brejvik and other neo-fascist terrorists are still online and are not facing any kind of repression.
The site lists actual terror attacks where the tools were used. That’s pretty damning if accurate. Anyone care to comment on that? I’m not well informed but the statement does not look like just straight up name calling.
> The United States Postal Service has spent months developing a U.S. Federal Ballot Mail Portal to provide election officials with a simple, secure, and efficient way to share lists of individuals receiving ballots by mail in their respective states.
Why? Your job is to transfer pieces of mail from one place to another. Just continue that. Everything else is manufacturing a problem where there was none.
Their job is whatever the government tells them it is, things like this don't just spawn out of thin air, there's usually rationale for it[0].
The post office already does way more than you think it does. Like identity verification on behalf of the government and tracking which houses are considered "vacant" or not.
It would be nice to focus the USPS, but all these extra services have to be done by _some entity_, otherwise you can't have the final result. In this case, the result is a functional, auditable mail in ballot system
> when Firefox does things to push away users like Mozilla buying up an ad-tech company
What would you have them do? Somehow they have to make some money somehow.
Alternatively they have to live purely from donations, which means that people like you and I have to part with of our money. Or they could be purely open source, in which case people like and I donate our time.
How is it that we can have a plethora of different Linux distributions, the Linux kernel, the FreeBSD kernel, the OpenBSD kernel, GIMP, LibreOffice and several other large projects that are run and maintained by donations and volunteers.
Yet somehow Firefox is different? It needs to make money! It needs to make money so much that they keep spinning up all sorts of new projects so that they can make more money, or at least that was the argument for some of what they were doing.
At this point in time, if they had taken all the money they spent on their various projects and put it in a trust they would probably be able to fund development without taking in any more money.
And it's not like people wouldn't donate to them either, look at how much Wikipedia has pulled in from donations.
> if they had taken all the money they spent on their various projects and put it in a trust they would probably be able to fund development without taking in any more money.
Mozilla Corporation already has funds totaling about $1 billion today. They're simply choosing not to develop Firefox actively.
We've got a live one. Most of their expenses are developing the core browser, the side bets (with a few exceptions, notably FirefoxOS, which I think was a GOOD side bet and worth the effort) were not that expensive. They're developing 40 million lines of code, and a lot of the things you're comparing it to actually aren't as comparably large or cutthroat.
>At this point in time, if they had taken all the money they spent on their various projects and put it in a trust they would probably be able to fund development without taking in any more money.
They already do that, they have a huge endowment set aside. It's obvious that people don't actually care about it because they don't know, and they only mention the possibility to make the accusation, never to give them credit once they discover Mozilla already does this and has been doing it for about 2 decades.
>And it's not like people wouldn't donate to them either, look at how much Wikipedia has pulled in from donations.
Another delusional argument. Wikipedia brings in a fraction of what Mozilla gets, and with orders of magnitude more users. And that's the best pure online fundraising program on planet Earth. Again, people only think this through to the point of being able to make the accusation not to the point of figuring out if it would actually work or not.
>They're developing 40 million lines of code, and a lot of the things you're comparing it to actually aren't as comparably large or cutthroat.
To get pedantic about it, they aren't developing that much code, they are maintaining that much code.
Not that I think LOC is the best metric but here are the figures for 2 of the project I brought up.
Linux kernel is ~40M LOC
FreeBSD kernel 7.8~9M LOC
I would argue that the OS's that run the internet are much more cutthroat than a browser, not just in terms of attack risk but in business terms as well.
>Microsoft aggressively tried to destroy Linux in the late 1990s and 2000s. This era included the internal "Halloween Documents" mapping out strategies to fight open source, former CEO Steve Ballmer calling Linux "a cancer," and Microsoft covertly funding SCO Group's proxy lawsuits against Linux users.
The larger your codebase is, the greater the attack surface. One of the arguments about the way that Firefox is progressing is that they keep adding more stuff into the codebase. If it's getting too complex, maybe they shouldn't be adding those features...
>Another delusional argument. Wikipedia brings in a fraction of what Mozilla gets, and with orders of magnitude more users.
Mozilla received 16.9 million from contributions in 2024.
Wikipedia took in ~170 million in the same time period.
Not sure where you got that Mozilla was bringing in more than Wikipedia through donations?
In going through the Mozilla Financials, I don't see where they have a huge endowment set aside, 73% of their revenue is royalties. I did however see that they spent 31 million to buy Acronym(an ad company) and 21 million to buy Fakespot. So that's neat.
It's not an either/or situation. For many years now people have been begging Mozilla to allow them to give money to help fund firefox (and only firefox) but Mozilla has refused. I can't promise that allowing it would provide all the funds they need, but certainly every little bit helps. People could continue to contribute code to the project, but that requires a lot more time, effort, and no small amount of skill compared to the time it takes to enter your credit card number and submit and an online form. Contributing code to Firefox is still a good thing, but it's a process (https://firefox-source-docs.mozilla.org/contributing/contrib...) that won't be for everyone.
There is no precedent for donations on the scale of what Mozilla gets from search licensing. The best online fundraising drive on the planet is Wikipedia with orders of magnitude more users and they don't raise half of what Mozilla spends on keeping up with Google.
It's not the panacea people keep assuming it is, it's just one of many accusations that gets flung at the wall.
Except Thunderbird now does run on donations (though is looking a relevant additional funding sources, e.g. providing hosted email: https://www.tb.pro/en-US/). This is not to say Mozilla can't seek out other sources of money, or develop appropriate ancillary services, but the absence of an option to just give money for Firefox (and Firefox only) is noticeable.
I really want you to stop and think about what you're saying for a second. I just made the point that the strongest online fundraising operation in the world, Wikipedia, gets less than half of what it takes to fund Firefox. Your counterpoint to that is to point to Thunderbird, which raises significantly less money than Wikipedia, which is already not enough to fun Firefox, in order to prove that it's plausible that the full operations of Firefox could be funded.
This would be the point where Logan Roy says you're not serious people.
I would have them fully shed the silicon valley tech corporation mindset and instead act like a charity working for the benefit of the public, as is appropriate for a non-profit foundation. Funding should mean seeking donations and government grants, not trying to fleece users, sell their privacy/attention and/or resell mediocre services - which are not serving the mission of building the best browser possible.
I am hanging on! At home I only use Firefox. At work, unfortunately, I need Chrome for a few things that do not work well with Firefox (most Google products), but for everything else I use Firefox.
IMHO, that Firefox can only really stay free if it is supported by donations (i.e. people like us who use it). Anything else will lead to various things that either push people away or make less free - probably both.
Could be. Pity there is no way to donate to Firefox. You can only donate to Mozilla for 'mission to keep the Web open and free', with an unknown fraction going to Firefox development. People have been deliberately disenfranchised from voting with their wallets for decades.
After 35 years in the industry in various capacities (from architect, engineer, VP, and different products, small and large companies, ...) I can heartily confirm this.
A company with a good culture is like a good football or soccer team. They work with each other to win, they know each other (enough for the job at least).
I am working at a bigger place now and the AI mandates handed down have been quite unproductive... As if we wouldn't have been able ourselves to figure out how to use the new tools - like we have since the beginning of software.
If management trusts you do to your job and has your back, then you trust the management and will naturally do a better job. Now there are always a few people who are exceptions to this, and part of good culture to identify those and remove them.
> I am working at a bigger place now and the AI mandates handed down have been quite unproductive... As if we wouldn't have been able ourselves to figure out how to use the new tools - like we have since the beginning of software.
This is so true. We have figured out how to use an endless list of software tools. Hardly anyone is as prepared to use the new tools as software engineers.
It's rather interesting that Apple (and Arm) are blazing the trail while Intel is seemingly left in the dust. Maybe not entirely on raw performance, but definitely on performance per watt.
Intel has been making pretty good progress on efficiency the last year and efficiency is the name of the game, you basically can't do performance without it.
My less-than-one-year-old Core Ultra 9 275HX was supposed to be much more power-efficient than previous generations, but I’m not impressed with it at all. Yes, it’s a gaming laptop, but why does it need 70 watts just to idle? And when playing Factorio with a CPU-heavy save, the PC uses 170 watts, while my M5 uses 50 watts on the exact same save. I don’t know how much the CPU alone is to blame for this, but I’m tired of x86 and Intel. My Linux laptop/server with Ubuntu and an AMD CPU is somewhat better (Also a gaming laptop), though. It’s only drawing 25 watts while idle while hosting six web services. So, if I were to go by just my anecdotal experience, having used about 10 different laptops over the last five years, Macs are just in another league when it comes to efficiency. However, the AMD/Linux combination is getting somewhat better at a slow pace, while Windows/Intel is stuck with terrible efficiency.
We knew about this all since the 1980's. Carl Sagan testified before congress in 1985 [1]. That's over 40 years ago.
Yet, we continue to burn more and fossil fuels; solar, wind, EVs, etc, non-withstanding. And right now some governments are actively spending money to work against renewable energy.
At this point I am afraid that we humans will only act when the alternative is unaffordable, when the forest is already on fire. I hope by that time it won't be too late.
Those of us who have been saying this all along won't even get the hollow satisfaction of an "I told you so!", because we'll be too busy surviving.
Vote, drive less (combustion engines), fly less, use public transportation, check your utility mix (and demand a larger renewable fraction), install solar, lobby, buy more local (where possible), consume less, support those with jobs displaced by renewables, accept some loss on your fossil stocks, ...
I think it's more a mix of anxiety, frustration, and exploration.
Exploration: Hey there's some cool new tech, let's see what it can do for me, for us.
Anxiety: What will happen to my craft? Will my skills still be important? Will I have a job in a year? How will I keep up? If we stop or slow hiring junior engineers, what happens when the experienced folks retire?
Frustration: This could be a time of fun and exploration with these new tools at hand. Instead we are measured by weird metrics (tokens, PR times, etc, etc), some of us are laid off, etc. There are $100bns spent on AI data centers.
The last one is the closest to anger, but it's not really the same.
More on the frustration: this person with no real world experience is hired and all of a sudden sending me six page articles they themselves couldn't fathom writing and they're lauded by founders and celebrated as pioneers of the company. Nobody in that chain could pick out a race condition from a lineup of cat feces. And I'm the "bad guy" for poking holes in all of the Groundbreaking Computer Science being done.
It's unbearable that most LLM inference happens on closed H/W, closed drivers, and closed SDKs.
reply