Setting aside incentives – because I think that's very fair to push back on, we should strive for better craft...
I find Blow in particular a bit naive when it comes to the complexity of modern non-game software. Games are very complex, but Blow puts them on a pedestal separate from other software, as if a web backend can't be as complex. He calls software slow while simultaneously ignoring most of the functionality.
He's essentially never had to deal with distributed systems, or anything that allows multiple users to use the same system, with all the performance and anti-abuse issues that come with that.
His opinions about game development are insightful, and his thoughts on craft can be quite inspiring, but I think he could benefit from a stint on a (good) big tech server team to see some of the things he's currently missing.
Heh heh heh. In actuality, Jon Blow has plenty of experience with distributed and multi-user systems. I worked closely with him while we were developing TinyMUCK 2.x, and the MUF programming language, a stack-based, simplified Forth.
In fact, Jon played a crucial role in the invention of the first MUF worm. We had just finished an extension to MUF that permitted triggering programs wherever a text string was expected. This gave a great flexibility and functionality to run MUF programs passively, by third parties, all kinds of new ways.
Unfortunately this also gave rise to a "worm" type of MUF that replicated itself whenever it was triggered passively. We went a little crazy when this was discovered, but it was all part and parcel of a multi-user game system, where there were permissions to consider, and multiple read/write/execute access to objects that should be owned by their respective player characters.
Jon Blow also inspired a lot of my exploits as I tested other systems of MUD programmability. In that time, TinyMUSH was also experimenting with a language that was more akin to scripting, and boy was it vulnerable. TinyMOO was also coming out, with a rather revolutionary programming paradigm. So in the mid-1990s, just in the insular TinyMU* world, there were at least 3 branches of programmable MUDs being actively developed.
Hopefully Jon could recall my greatest contribution to the corpus of early MUF programs. On Atlantis MUCK, I programmed a simulation of Conway's Game of Life. Of course, there was no animation possible and there was no multithreaded execution possible, either. So my program was given a starting board of Life, and then computed each frame in sequence, output it into the text stream of the game, and terminated after N iterations. Meanwhile, the entire game server locked up, because it could only run one MUF program at a time. Everyone else's commands and input would be held until my MUF program finished running.
This was a real issue, of course, and Jon and I worked to mitigate the problems caused. I think, ultimately, we just limited the number of iterations or the number of instructions that could be executed by any one MUF program at a time.
But Jon and I collaborated on a tutorial for player-programmers, and he helped us find and extinguish many bugs. And these were very complex systems, with 64 or even 256 simultaneous players.
Perhaps Jon has forgotten some of those lessons, but perhaps not.
Fair enough! That's some good experience and should address some of what I've heard from him, but perhaps he has forgotten.
That said, and this is coming from someone who (sadly) missed the era of computing you're describing, I think there's still a difference between dealing with a 64-256 player multiplayer system, and scaling a server application across dozens? thousands? of machines, across the world.
My limited research suggests that MUDs run almost exclusively on a single machine (?), or even just having a small number in one location drastically simplifies the problem space.
It's hard because you have to constantly make "sub-optimal" decisions. If you optimise for things you can measure, or at least see right now, you get a low performing team of high performers, or you get a collection of local maxima. Optimising for a global maximum requires making decisions that have unclear pay-off, that are harder to justify. Another problem is that one of these decisions that was actually wrong, vs one that hasn't paid off yet, look the same.
I suspect executives very much get it, but realise that there's no obviously good answer here.
Part of me wonders whether trying to optimise at all is the problem.
The worst managers I've had were optimisers (and had the worst teams), and the best managers with the best teams weren't optimisers.
The best ones didn't care about metrics at all, but would work to get their teams the tools and/or conditions they wanted to do a good job. Of course it probably only works with a certain level of talent and alignment in the team
There's also the executive's incentives, and "I'll be gone, you'll be gone": the payoff for that is very long, and it will be hard for them to take credit, and their own tenure might not be as long as that team's.
There's one caveat to this – I think it's good to have bias to other people's actions early on in a new role. Taking on little projects that are already understood, taking on cleanups, things other people aren't volunteering for, is all a great way to find where some of the bodies are buried, understand how the team works, and prove yourself as a team player.
Which British businesses provide this sort of infrastructure?
I ask this as a Brit who ran stuff on a few UK providers, and ultimately moved to a cloud platform because of the combination of terrible service and lack of features. The landscape may have changed, but in my experience it was bleak circa 2020.
There's a now dead comment giving a few options. One of those options was the main driver for my previous company moving fully to GCP, they were atrocious and caused a major security incident for us. They also regularly took days to action urgent fixes.
I'm not saying the UK doesn't have data centres, but it's easy to do a crap job of running one. In my experience there's a level of competence that is missing from UK operations. Hetzner and OVH are noticeably better, but still not quite at the level of the main cloud providers.
Nice demo, but can we talk about how this is patented. Should it be? Should something like this actually get granted.
While it's neat, it's neat on the order of a UX idea someone might just come up with at standup one day. Like, cool, worth an A/B test, but worth a patent? I don't think so.
I thought the same thing, just opened it and went "patent pending... Really?"
I am genuinely annoyed when someone gets a helpful UI idea and their immediate thought is "wow, I should patent this so I can monopolize this helpful idea".
> Didn't even need to talk to anyone to port my old number
That seems like pretty low stakes – I wouldn't expect to talk to anyone to sign up and transfer to a consumer service in the UK. I don't think I've spoken to anyone for something like this in the last 10 years or so.
I've never talked to someone for it, and I would've first done it in something like 2008. (It's 'always' been done by SMS, well before 'apps' that might let you (and I don't recall it on websites).)
Well I haven't moved my number for a couple of decades at least, but the last time I did you had to phone them and go through the whole "are you sure? Let me see if I can get you a better deal" nonsense.
Cancellation is different to porting a number. You'll still need to cancel your previous plan, but legislation fixed carriers holding numbers hostage a while ago.
I can't help but think this isn't the best idea, it raises so many questions. How does fail over work (it failed for me!), can you safely fail over, how does product think about the stack, is it funded well enough, how do you scale a system from zero to all traffic instantly, how do you effectively keep this environment warm, how do you determine which services are critical enough to be in it, how do you manage finding the bugs in two separate stacks, and so on and on and on.
It seems like it would be better to invest in a platform that is distributed across multiple failure domains. Some companies call this a cell architecture. For Monzo this might mean true multi cloud, but they already have bare metal services for bank integrations so they are already hybrid in some respects.
It's not the decision I would have made, and I'm not sure I agree with their reasoning, but I'll also admit that I have not run a bank, I don't know all their constraints, and there are some good tech folks at Monzo. That said, I am an SRE at Google so I'd like to think I'm not completely naïve here.
Edit: after reading in more detail I'm puzzled. The justification for Stand-In and not traditional scale-out is the difficulty of data consistency (agreed!), but then Stand-In is just eventually consistent anyway, and introduces sync problems in both directions. Admittedly I'm spoilt by Spanner, but I do wonder if Monzo is being bitten by past architecture choices here.
Fwiw to acquire a banking license and get the relevant approvals I believe you need to prove a redundancy plan that goes beyond just scaling.
I used to work in fintech where we’d face the same requirements and know a bunch of folks from Monzo. You can fight the status quo only for so long until it becomes much easier to just do what you’re being asked and produce a totally separate service that solidly answers any audit you may face.
For truly critical systems and independently developed redundant or backup system that can perform at least the most important functions is clearly superior. Regardless of how much effort goes into making one implementation reliable, there's always the possibility of an issue that takes it out, or forces it to be taken offline.
I don't know how common this is in retail banking but I'm happy to see Monzo do it.
I'm a bit curious about Spanner. At $EMPLOYER we were going to do multi cloud with multi-region CockroachDB clusters but gave up on it because write latencies and costs were too high. Do you not get the same thing with Spanner, where technically you can have multi region but it's very unattractive to actually do?
I'm afraid I'd have to defer to the public docs on this. I've only used it internally and a bunch of that is not public.
What I would say is that it's considered to be much better than CockroachDB, but that there's no getting around physics. Spanner is subject to the same speed of light... but in every other way it does what it can do to improve things at every level. Google's use of GPS hardware for better timing is public.
monzo r the same guys who went in on micro services. at a time they were bragging 3 micro services per engineer. so yeah a lot of things don't make sense.
No, it's a great idea. Any systems engineer knows redundancy is good, even better if it's across providers. I see no issues with this from a system architecture point of view. As to how the implementation is done that's up to them to do right.
Redundancy is great, absolutely, but there are multiple ways to do it and this is not necessarily the right one.
Which is better: 3 DCs for 2 DCs worth of traffic, running the same stack but isolated failure domains... or 4 DCs for 2 DCs worth of traffic, that work in completely different ways, where 2 are left cold until needed. I can see some advantages of the latter, but not a lot.
This also isn't about fault tolerance at the code level like in, say, avionics. In those scenarios having multiple distinct implementations makes sense, but that's very different, you run all paths at the same time and check they agree etc.
I've never run a bank either, but their current setup is very cost effective. A true multi cloud architecture would be "much" (a couple of 9s) more available, but I believe it would introduce other issues. The goal was to ensure the use of a subset of features for short periods of time, which is probably already above the market standard.
The cost effectiveness doesn't really add up for me either though (and again, I lack a lot of context). 1% of cost to run everything essential doesn't pass the sniff test, I assume it's much more during an actual failover. But I'm also not suggesting running any more hardware than the number of failure domains you want to be resilient to losing, and on cloud you could choose to run less and scale (assuming you aren't concerned about stockouts).
My guess is a little of both. I would bet that the 1% is the unused cost, but that for periods that they are running the stand-in system it's much closer to normal. But from what I've heard of their infrastructure I'd also bet that stand-in is significantly cheaper in part because it had the benefit of hindsight and could resolve issues in the first system.
None of this really changes my original feeling though. A cell based architecture spreads the risk, failover is essentially built-in without being a special mode you have to put the system into and then recover from again, on cloud you can scale those up and down with traffic and they also take peaks from each other, and so on. There's a reason why there are very few cloud outages that cross all regions for a provider, and if you're paranoid you can run multi-cloud (as they already do).
"Just do multi-cloud" is burying the lede a bit. Monzo is a bank, so it's not a given that they can easily partition all their operations into fully independent cells. I'm reasonably confident that teams of average competence could implement a second system for failover. I'm much less confident in teams of average competence doing complicated distributed systems work correctly, and doing it incorrectly is potentially existential risk for a bank.
In your Google scope this all works out however if you're a mature but not yet prevalent, neobank whose reputation rides almost solely on availability, I'd still say theirs is a great approach. Even in the unlikely event that AWS throws in the towel and has a hard failure, users can still enjoy a subset of functions without elevated latency. On top of it, it might well be that user data can never leave UK, so geographical redundancy might not even be a thing here.
And yes you are completely right a completely distinct implementation is probably a no-go, but I'd think you could deploy a similar codebase with feature flags, with some frontend differences. I'm speculating but pretty sure they didn't reimplement backend functionality for this.
Agree, I know nothing about their system but in terms of clear auditability and accountability of systems I agree it's great. Every subsystem failure or inconsistency can be clearly attributed to one or the other site/stack, and the stand-in could for example apply other rules (e.g. approve only smaller transaction amounts that are accepted by an incident-only set of risk rules).
I suspect Monzo was bitten by complexity. The network should not be assumed to be reliable and if you make a thousand things (microservices) with network between them regardless of what you think may happen, it’s entirely impossible to test it with any level of certainty. And that’s worse if you have a complete failover scenario.
Add the instability of the average cloud control plane during an outage and things get rather nasty rather quickly in reality. We’ve been through that with AWS’ numerous outages.
My company is victim of this mindset. We sacrificed tested and assured reliability on physical hardware in multiple redundant data centres for cloud and microservices. Mostly driven by Monzo’s claims, semi religious thinking and me too.
Reliability is measurably down, cost is higher, latency is higher and we cannot fail over to another provider because someone went to an AWS conference and started using all the toys. Even if we did get rid of those dependencies, during an outage the kubernetes control plane is so tied into the vendor that we’d be fucked anyway. Last time the ingress controller couldn’t change ALBs for example.
Only winner here is Bezos and the decision makers CVs. The losers are the customers. We live in the stupidest of times.
I think the biggest problem is the data consistency. Since they are planning for eventual consistency, should make it easier, but they most probably had to re-write code to support atomic operations and proper transaction isolations.
But the scaling problem is still there, when they switched over I assume they would start both horizontal and vertical scale of k8 pods and maybe even bump up DB instances as well which could result in a small congestion at the start of the switch over
Foundationdb has a good solution to this. Uses three regions, sync writes to regions 1 and 2 which should be near each other, sync writes to region 3. If region 1 dies then region 3 pulls last commits from region 2 before serving traffic. Small cluster manager runs in all three regions
So I've been spending a lot of time in the financial services/payment space and might be able to answer some of your questions.
The EU activated the Digital Online Resiliency Act (DORA) last year (passed in 2023) which, amongst other things, mandates that financial services providers who run their core business functions in the cloud (i.e. Monzo) must be able to demonstrate that they are not entirely reliant on a single cloud provider. Otherwise, big, big fines.
Like GDPR, there are auditors that are responsible for signing off on EU DORA compliance. Also like GDPR, there are ways to check those boxes without _really_ checking them. This is probably why Stand-In only has 18 services in its GCP failover "region" instead of the 1K+ production services in AWS; it's just enough to keep critical services (payments/card networks, balances, ledger, etc.) operational and, more importantly, keep DORA auditors at bay.
This article IMO is more for other FSIs who _want_ to run _entirely_ in the cloud (there are very, very few that do). Financial services is a very risk-averse industry. Most providers have biz-critical systems running on mainframes still. (Mainframes are practically indestructible and are much more technically-interesting than people give them credit for, but that's another post.) Being able to tell AWS "yeahhhhh we can run this business on GCP and/or Azure in literally 10 minutes" is a money line for procurement to use during price negotiations. Or at least it was until RAM prices mooned hard enough to make their existing datacenter inventory suddenly very attractive.
Cloud-aside, pretty much every bank in the US has some architecture like this for their critical banking systems, except they are usually an old-school hot-cold (sometimes hot-hot) datacenter failover strategy. Historically failover has been done manually with a runbook (with some automation) and tested during DR/BCP days (sometimes more frequently). They, of course, have zonal/regional replication in place.
It kind of makes sense that critical financial infrastructure would be one of the things that doesn’t just run on AWS and accepts it goes down with everything else when us-east-1 goes down
The second one is pretty much directly a Die Hard parody – new building, party, execs off doing drugs, dead guy in the lift. I'm looking forward to that one, I wouldn't be surprised if there's a lot more in there.
It looks like Rockstar have nailed a solid 2026 update to previous releases, everything looks great. But is that enough? It should be, but given the billions spent on development I think there's an expectation of more.
It also looks like they've kept the art and writing fresh for 2026. GTAV feels dated at this point, no criticism of course, and I was concerned that they would fail to modernise into a world that is quite different now than 13 years ago, but it seems they've likely succeeded at that.
I'm looking forward to playing, eventually, when it comes out on desktop. But is this the game of a generation that justifies buying a console? I'm not sure.
For sure its enough. We haven't had a GTA game for a decade.
There was some interview with the Skyrim people who didn't want to do the next skyrim because of the success of Skyrim. Lets be honest, just make another Skyrim a little bit better here, a little bit better there, new stores done.
A game is better than no game especially after that much time
I think it's mostly about the characters. GTAV was, in my opinion, much lighter-hearted than this is looking. Trevor was a hoot, Michael's marital problems were made light of, even Franklin as probably the most serious character still had plenty of quite direct comedy written around him.
I'm not sure that would go down well in 2026. The redneck thing feels tired, the public narrative of wealth inequality would likely make Michael feel out of touch, and much of the audience would have identified with Franklin but he doesn't really have the struggles that today's Millennials and Gen-Z have.
These two characters seem more realistic in that sense. Less direct comedy (although I still see that there's the iconic GTA parody in places), a couple who are probably going to be much more relatable even if they're criminals, and a standing in society that is more relatable.
It's not that the comedy wouldn't land, the parody/satire is important and it looks like it's still there, it's more that I think V depended on some stereotypes that are long gone, or that society recognises differently now.
I don’t play games to find people I relate to. I play games to become a character for a period of time in the game. Rich, poor, redneck… it doesn’t really matter. They can all be interesting and engaging characters to take on if the game is written well and fun to play.
I think this a mistake a lot of modern game companies are making. They think people want to be represented, when what most people want is an escape. I want to do things in a game that I would never be able to do in real life. That’s half the fun.
I also think the venn diagram of people offended by stereotypes and those who play GTA barely overlap. They may be completely separate circles. The people who will complain about the game were never going to buy it, so Rockstar shouldn’t care about what they have to say. Make the game for the customers, not the activists and critics.
Let's say I'm caucasian male grown up in poor ex-USSR shithole struggling through immigration, changing countries and losing social circle multiple times along the way and struggling with loneliness, etc.
Then the least I want to do is to play game as migrant from Eastern Europe. Because if the character and their story is accurate and representive then I just be bored from it. And if it's inaccurate then it's just bad story telling.
I The Office, Dwight made a version of himself in Second Life that was the same as his real life. The whole idea that someone would want to experience their actual life in game form was the joke.
I see your point, and maybe I'm over indexing on this or not describing it well.
I don't see myself in either of these characters, but I do see them as representative of society, whereas I don't see the GTAV characters as representative of society. However, I think I did when I first played GTAV the year it came out.
The last game I played was James Bond First Light – and that's for the escapism you describe, I got to "be" James Bond, great. But in the same way, the game and story felt relevant today, with aspects being believable in 2026, which is a stark comparison to how watching Thunderball recently made me feel – that was jarring and much harder to "escape" into because of how out of touch it was (obviously understandably so).
I get what you are saying, I however just don't agree.
GTA has it roots in a story about the criminal world in a version of a US city with a clash of many subcultures within it.
I don't see at what time they tried to cater to teens, outside the parodying themes of that generation which they fit perfectly. I saw that every installment representative of a criminal from one of these cultures to top of the criminal food chain.
Be it the Scar Face parody of Vice City, Gang culture of San Andreas, the Immegrant story of the 4th or the trio they went for with the 5 with the LA story melting pot.
As a none-american it perfectly represented the characters and views of over the top depictions of the movie version of these places.
> Be it the Scar Face parody of Vice City, Gang culture of San Andreas, the Immegrant story of the 4th or the trio they went for with the 5 with the LA story melting pot.
I think this is the crux of my point actually – these are all stories that were more relevant at the times they were released. GTAV seems to be more down to earth, and modernised the stereotypes that it's poking fun at.
Gangsters don't resonate right now (in my opinion), immigration is viewed very differently, wealth inequality is viewed differently.
> As a non-american it perfectly represented the characters and views of over the top depictions of the movie version of these places.
Maybe we disagree a little here – also not American, but where I feel that GTAV was more of that over the top depiction of the movie version, I'm getting a different vibe from GTAVI, something a little closer to the real America we see today.
> I'm getting a different vibe from GTAVI, something a little closer to the real America we see today.
It seems like they are adding a lot of references to viral videos and news clips into the game. I’m an American, and while these things happened, they are all exceptional events, that’s why they made the news and went viral. Day-to-day, these over the top things aren’t things most people see all the time. The game seems like it’s going to take the “Florida Man” stereotype and turn it up to 11… as a general vibe. I don’t know that we saw enough to get a good read on the main characters, at least I haven’t.
I think it’s easy to get a skewed perspective of what real America is like from the news or bodycam footage… but it is entertaining, which will hopefully make for a good game.
I mostly play games that have good gameplay mechanics, be it fast action games, strategy or racing sims. There are very few games that can even get close to movies regarding characters and story, so I don't bother. That includes GTA with the possible exception of part IV.
Bioshock Infinite, for example, doesn't do it for me. It's a pretty bad shooter and the story is not as good as its authors think it is. The Witcher 3 doesn't do much for me, but I will try again after the recent free update - supposedly it's got improved combat.
The only game with not great mechanics that I like so far is Mass Effect. I'm impressed by the characters that feel like real-actual people and the varying environments that all feel true to life, all following the logic of their incentives and resembling their parallels in the real world. For example, the free trade moon(?) feels like something between Switzerland (or Singapore) and an office skyscraper, but with a little more crime and corruption - the local government just wants business to run smoothly and is willing to overlook some plausibly deniable sins of good customers. I have never seen anything like it in a game.
What are some games in various genres that you consider to have good gameplay mechanics? For me, I'd have to say Resident Evil 4 (original, haven't played the new one yet), Smash Bros. Ultimate, Rocket League, and, weirdly, Pikmin 3's mission mode.
Definitely Rocket League, the new Doom games (upgrades and glory kills are out of place though), Brutal Doom, racing sims like Dirt Rally and Assetto Corsa - difficult as heck but lowering expectations kinda works, the Half-Lifes, Hollow Knight has strong mechanics (I like the atmosphere but not the messy backstory), Command & Conquer 1 (far and away my all-time favorite for style, great units too, AI is trash), XCOM though the pod mechanics are a major blunder, the suspense and problem solving in Splinter Cell is nice (kinda clunky controls but movement is supposed to be difficult), Broforce has silly and satisfying mechanics, X-Wing and TIE Fighter (again)...
I recently got the remake of Resident Evil 1 and oh boy, the clunkiness of just about everything is off the charts just like I remember from a friend's PS1. I did want to like it, but somehow, Japanese games with a serious tone never seem to do it for me.
The gaming world, which is what I presume they meant, is much different today. The days of clunky gunplay and insta-failing a mission for going 10 feet out of the correct path are long gone and would be tiresome for most gamers.
Absolutely, I replayed some of GTAV is ~2018 it already felt a bit clunky in that respect.
I think some of this is technological, but much of it is style. There are likely limitations that forced some of those issues back in 2013, but I think it's more down to game design practices. It doesn't take a lot of tech to give you another 10 feet of leeway and trigger a sound bite with a snarky comment from another character.
Wait till you replay the earlier games. I bought the definitive edition trilogy in a sale, only just started playing thru them. Decided to skip GTA III completely after seeing the reviews and lack of unofficial patches. Vice City feels much more playable after adding a bunch of patches, but unfortunately it doesn't fix the clunkyness and difficulty of the missions. I'm only like 7 or 8 missions in and feel like quitting, having to retry several missions multiple times. Especially that RC helicopter mission, I've no idea how I even managed to complete that (took like 6 retries). But I'll somehow try and chug thru it.
Hopefully San Andreas is lot better (and given the number of mods available for it, it feels like it would be).
> The days of clunky gunplay and insta-failing a mission for going 10 feet out of the correct path are long gone and would be tiresome for most gamers.
Let's not kid ourselves that either of those things is going to be any different in GTA 6.
Just from the video you can gather that there's much more than just a visual upgrade. The fact that you can apparently just rob random stores, that there seems to be specific ways to open cars (so different cars, different difficulties?), the variety of activities, and the pnjs that seem to remember you, etc.
I was afraid it would not live up to the hype, but now I think it just might
As someone who finds Rockstar gameplay mid, deep down want GTA6 fail and Rockstar turn into engine company like unreal. I know big part of the formula is $$$ but there's so much neat stuff in Rockstar games that AAA hasn't gotten close to in 10 years and I sense won't.
I know that's the likely timeline. My hope is that because consoles today look a lot more like PCs than they did in 2013 (or earlier during development), that it might be sooner, plus from leaks we know that development appears to mostly be happening on PC (I mean for real builds not just engine tools and cross-compiling).
But I realise that's just hope. It's probably also about business priorities and PC is just not a huge market.
Right now AI has trouble maintaining coherence over a few paragraphs of text. How long will it be before it is capable of maintaining coherence over something the size of a GTA game? And how much money will that cost?
Where people got this billions cost from? I doubt that they spent that much in GTA VI. RDR2 gave the base for GTA VI (engine and things like that), so I just find really odd they would spend 2 billions on the rest
They've themselves said they've at least doubled in size since RDR2 for this game, with a studio, for example, SOLELY dedicated to random crowd NPCs systems(!)
With the average Sony game with a fraction of this game's features costing upwards of 400mill, don't be so sure this behemoth didn't hit the 1.5 to 2 bill mark. Paying a growing number of devs, testers, plus insane amounts of marketing push for over a decade adds up more than you may think.
Other less complex recent AAA games cost 300-500 million, this game is easily between 1 and 2 billion, specially if you include marketing.
And just GTA Online is making 10 million USD every week.
The game screams money just looking at the sheer density of the city/map and the level of detail. There’s simply nothing quite like this, as usual with Rockstar.
reply