I am also very glad for Meta's investment into the future, so second your voice here. In a sense though it's Palmer Luckey who punched the tech into a defensible product. John Carmack surely tried to do the right thing, but I think part of why he left is that it's too difficult for any individual to make significant (risky) design decisions.
Do I have faith in Meta to 'win' in VR or AR commercially? Well... not really, but the technological advancements are real.
The AI generated character is a weighted average of many characters written by humans.
Which are not created from nothing. People write characters based on a combination of other fictional characters, real characters, and perhaps some 'RNG'.
Seems the distinction is that the AI generated character can not have any direct bearing on reality, because the LLM never got to know anyone directly. Not derived directly from experiences rooted in reality.
I fail to see how they will take a significant market share or even break even on this venture.
It's an overcrowded market. Far better would be to focus on semiconductor supply chain, which Japan already supplies some elements, to sell to fabs.
Because the bottleneck is the fabs. If a new 2 nm fab came online today, it would immediately sell all its capacity to 2030 no problem, without Fujitsu trying this gambit.
It's not 'sovereign', the architecture is not designed in Japan, the silicon is not fabbed in Japan. This whole thing is sideways.
Previous generation of this was a monstrosity based on a custom SPARC with ARM front end grafted on top, supporting the version of SPARC extensions they created backported to ARM as a new extension.
So there's a good chance that it's still an optimized half-SPARC inside.
You have to load the model weights into VRAM over PCI-E (from RAM). So the (PCI-E) bandwidth strongly affects time to first token.
You have to run inference on the GPU by reading and writing to VRAM. So TFLOPS of the compute matters, and bandwidth to the VRAM (Always integrated with the GPU, rarely a bottleneck), and this strongly affects tokens/s
If you're doing training workloads or offloading to system RAM, it gets more complicated. (And mostly bound up trying to feed compute on time)
> So the (PCI-E) bandwidth strongly affects time to first token
On dedicated inference hardware I'd expect model weights to never leave the RAM, and you'd probably load them on startup before even starting to serve requests
There are deeper structural problems when you look at the incentives inherent to government driven society.
While I do appreciate and even support the idea that the immense wealth available on planet earth should be more evenly distributed, and people should mostly be focused on raising families, deepening their philosophy, and perfecting their craft... (some societies in Europe are pretty laid back in terms of hyper-productivity culture)
Regulation is possible without paralyzing industry. Laws about the internet are possible without stifling freedom. Central planning is possible without crashing the economy so bad you end up with a regime change. It all comes down to execution and for the most part, it's been a series of failed experiments.
As much as I want a happy future for Canadians and Europeans, I just don't see the odds being super favourable.
Dunno. My impression is that with the EU, we pick its suboptimal parts and compare them to what they could be with optimal execution.
We don’t look either at the successful parts, not at the world where it is wholesale gone. My gut feeling is that EU members are, on the whole, much better off for being in than out. One natural experiment is the UK - opinions differ, but the drastic accumulation of problems coincides with Brexit timings.
I’m all for improving the EU! And for sure there are bad bits. But when your car AC is doing badly, you don’t write off the car. You try to fix the AC…
People sometimes call EU the sleeping giant.
Big potential compared to the actual results.
I'm more aware of what Canadian government does with its tax dollars.
And I'm just assuming that the EU is similarly quite expensive for what it achieves, and the budgets only change in one direction.
EU budget is actually quite small - about 1% of GDP.
That’s very small compared to EU members states national budgets, which sit at around 35% of GDP, apparently. Canadian federal government apparently spends 18-20% of Canadas GDP, topped up by local governments.
This is all very apples-to-oranges but doesn’t support the thesis that EU budget is particularly bloated.
Manny comments I agree with - it's a mistake, it's a reason to leave the distro...
I'd like to point out that Ubuntu/Canonical is now seemingly philosophically separate from the origins of GNU and Linux. If you're curious, read "Hackers" by Stephen Levy. They've lost the plot, so to speak, about the core spirit of open software and hardware, which is what gave the projects life in the first place. A philosophical understanding and unity between many, many top notch independent developers.
Another note is how AI contributions to such libraries and programs is going to have an unknown effect on quality. It's almost like there's a business case to rip all the good FOSS written by humans out of the hands of github and apt, and curate all the best source code to ensure it remains in circulation, and extant copies are available that are not washed out by loose standards WRT AI contribs. Or if not a business case, perhaps a reasonable reaction and a good personal vendetta.
> "until the context rot and sampling problem is fixed forever"
I agree, prompt adherence seems to get worse when operating on large inputs.
Does anyone have some notion of the SOTA with this? Can we expect big improvements by this time next year? (hopefully in open weights)
A lot of this is managed by the inference engine, and has nothing to do with the model.
Models that use, for example, sparse attention mechanisms are just trying to make the bad situation slightly less bad, such as using less RAM for context (thus requiring less context quantization) or using less bandwidth (thus running faster).
If people keep using temp, top-k, top-p, and min-p, and nothing else for samplers, we're ignoring ~3 years of sampling research that virtually eliminates the worst of context rot issues.
Your hardware can do way more than 64k tokens context window, can't it?
And with Ollama it's very easy, superficially you just drag the slider.
I'm now reading "Friends Don't Let Friends Use Ollama" linked in another comment so a lot of problems with that approach are surfacing for me right now.
So yeah. Along with others, I think you should come up with some empirical means of understanding if your preprompt is doing anything good since I doubt that it's all necessary and helpful. Second maybe you and I need to fix our runtimes.
Seeing the raw CoT is the best signal for evaluating prompt efficiency. The fact that frontier providers don’t share this feels criminal after this last weekend. They’ve played us all for fools.
Thanks for the token setting observation. I’m running ollama headless and from the command line. I didn’t even know there was a web ui. Conf file is pretty straightforward though.
I settled on 64k tokens because of a recommendation from an LLM. I now see that that guidance is associated with similar sounding (but not the same) configuration constraints. So this weekend I’ll try turning it to 11 (250k tokens?). Anticipating some hard locks over here.
CoT is performative and doesn’t reveal how reasoning happens. If you look at those traces locally it’s just gibberish, especially if the model falls into a loop.
This is good guidance, but what do you have to say about convincing your team of developers to live it out?
I've found that developers usually like writing code and avoid contributing to documentation. For some, it's actually scary because (edit: for them,) high quality writing is harder than high quality coding, and it can be avoided quite a bit.
On the project side, it's rare for the implementation and verification stages to not consume all the budget and more, and delivery creeping past the original optimistic date. So there's no time or money to spend on documentation.
The combination is that even with your great advice in hand, it's hard to navigate to really solid and comprehensive design documentation underpinning the products.
This is a good question, and I have a super long answer that's been in my head for like 8 years about how to influence your teammates to adopt good engineering practices.
The short answer is that most useful software engineering practices are a risk to the first person on the team to adopt them. For example, if everyone on your team thinks automated testing is stupid and you adopt automated testing, it will look like your work is worse because you're slower in the short-term, and maybe you have to do even more work when teammates break your tests.
It comes down to accruing social currency with your team. Your teammates don't want to take a risk for you if you have a history of bad ideas that wasted everyone's time. But if, for example, you implemented automated deploys to replace a tedious workflow developers had to do manually, people would see how your ideas have payoff, and they're more willing to invest a little bit if they expect ROI long-term.
When I've convinced my teammates to invest in design docs, I made sure I had some wins under my belt before I started pushing for everyone to write design docs. I invested a lot in docs myself so my teammates could see the value before I asked them to start writing.
This is also a place where you have to think about politics a bit. Documentation has a much better shot if it has support from the top, so think about the pitch to your manager or dev lead about how design docs make their jobs easier.
If you've not read Fred Brooks MMM, give it a shot.
It seems like you're implying that teams of 60-80 developers should be expected to outperform teams of 12. This is simply not true. The most important feature of source code as a language is that it allows precise mindshare among close knit teams. It doesn't guarantee it, but it makes it possible for people to talk about the product at a level that is otherwise very difficult.
A convenient side-effect of the source code is that it instructs the machine what to do. But instructing the machine was never the bottleneck, the essential difficulty of software development is in understanding what are the correct instructions to achieve some objective, not typing them out.
The problem is that communication doesn't scale at all. Having just 3 developers with good alignment about mental models, best practices, and design direction is hard enough, and if you found the right three people at the right time with the right ideas, you could generate billions of dollars of value.
Large monolithic teams on the order of 80 are a product of people in control not understanding how software development works, and how to make it work well.
Do I have faith in Meta to 'win' in VR or AR commercially? Well... not really, but the technological advancements are real.
reply