The Cost of Software Failure in an Era of Vibe Coding Across Different Domains

The 45-Minute Mistake

On August 1, 2012, the trading firm Knight Capital rolled out an update to eight servers and missed one. When the market opened, that server ran old code and began buying and selling shares wildly. In about 45 minutes, the firm lost roughly $440 million (the SEC later put it above $460 million) and was bought by a rival soon after.

Now imagine a to-do app showing the wrong button label for a day. Same kind of bug, wildly different bill. That gap is the subject of this essay.

In early 2025, Andrej Karpathy named a new way of working: "vibe coding." You describe what you want, an AI writes the code, and you go with the flow, often without reading every line. For many tasks that is wonderful. But AI made writing code cheap, not failure. The price of a mistake is still set by the world the software lives in. So let's tour fourteen kinds of software, then ask what happens when vibe coding walks into each room.

Four Questions That Set the Price of a Bug

To compare fields, "cost of failure" is really four questions.

1. How bad is one failure? A typo on a menu is not a lost life.
2. How far does it spread? Engineers call this the blast radius. A bug in one customer's account is small. A bug pushed to every machine on earth is not.
3. How fast do we notice? A crash is loud, and a loud failure is half fixed. A wrong number that looks right can hide for years.
4. Can we undo it? A bad web page can be rolled back in minutes. A bad rocket cannot be un-launched.

Think of a kitchen: burnt toast gets thrown out, but undercooked chicken at a wedding for 400 is a different problem. Same cook, different stakes. A field's habits mostly come from how it answers these four questions.

1. Aerospace: Assume Everything Will Break

Aerospace is the strict end of the spectrum. Every line of flight code must trace to a written requirement and a test, and the code is deliberately boring. The reason is history. In 1996, Ariane 5 blew up about 40 seconds into its flight because a number in code reused from Ariane 4 grew too big. It had a backup computer, but the backup ran the same software and failed the same way. That is a common-mode failure, like keeping a spare key in the same pocket as the first. Even independently written versions tend to fail on the same hard cases, as a 1980s experiment by Knight and Leveson showed. The 737 MAX crashes, which killed 346 people, added a harsh lesson about relying on a single sensor.

The takeaway: when mistakes can't be undone, you pay heavily up front.

2. Medical Devices: The Paper Trail Is the Safety Net

In the mid-1980s, the Therac-25 radiation machine gave several patients massive overdoses, and some died. A timing bug appeared only when operators typed fast, and most of the physical safety locks used on older models had been removed. Today, standards like IEC 62304 and regulators like the FDA require teams to document every risk and how it is handled. It is like a hospital pharmacy: labels, double-checks, and batch numbers, slow on purpose.

The takeaway: here the process is the product. You must show why you believed it would work.

3. Cars: A Computer You Sit Inside

A modern car holds well over a hundred small computers. The standard ISO 26262 sorts every function by how dangerous its failure would be: light care for the music player, heavy care for the brakes. Like a house, you inspect the gas line harder than the paint. And in a well-known 2013 court case, experts testified that Toyota's engine software was tangled and hard to verify, and the jury found against the company. Toyota settled soon after, and an earlier NASA review had found no electronic cause, so the case is still debated.

The takeaway: put the strictest rules only where the danger is.

4. Power Grids and Industrial Control: The Silent Alarm

In August 2003, about 50 million people in the northeastern US and Canada lost power. One cause was a software bug that silenced a control-room alarm, so operators never got the warnings. It was a smoke detector with a dead battery: everything looks fine until it isn't. Industrial systems also live for decades, so updates are rare, and proven, dull technology often wins.

The takeaway: when systems run for decades, predictable beats clever.

5. Banking and Payments: Every Penny Must Add Up

Banks care about getting the books right. Money is stored in exact decimals, every payment is recorded twice (out and in) so the books check themselves, and operations are repeatable, so a payment sent twice is charged once. It is like a restaurant bill when the waiter's tablet crashes: you pay once, not twice or never. When it goes wrong, trust is the casualty. In 2018, TSB moved its customers to a new system, and around 1.9 million were affected, many locked out for days or weeks, at a cost of hundreds of millions of pounds.

The takeaway: when the product is trust, big, sudden changes that can't be undone are the danger.

6. Trading Firms: Speed With a Seatbelt

Back to Knight Capital. The problem was not a clever algorithm but a deployment mistake: a new feature reused an old switch, and the one server missing the update woke dead code. Nothing stopped it. The lesson was to assume code will misbehave and build a seatbelt outside it: checks before every order, hard loss limits, and a human kill switch. Like a race car, you want speed, but the roll cage works whether or not the driver is having a good day.

The takeaway: when losses compound by the second, rely on limits that hold even when the code is wrong.

7. Quant Research: The Bug That Looks Like Success

Quant firms use data to find patterns in markets, and their biggest danger is a result that looks wonderful and is false. A strategy might peek at tomorrow's prices, skip companies that went bankrupt, or be off by one day. No error appears, and the code tells a beautiful lie. It is a student who found the answer key: perfect score, zero learning. So quant firms spread their bets, set loss limits, and assume some of their "edge" is not real. Long-Term Capital Management's models worked until the world did something they had not seen, and heavy borrowing turned it into the 1998 near-crisis that needed a Wall Street rescue organized by the New York Fed.

The takeaway: the most dangerous bug gives you the answer you hoped for.

8. Crypto and Smart Contracts: Code Is the Law, and You Can't Edit the Law

Smart contracts are programs on a blockchain that often control real money and usually cannot be changed once deployed. In 2016, an attacker exploited a flaw in how The DAO handled withdrawals and drained roughly $60 million. The code did exactly what it said, just not what its authors meant. (Ethereum later split its own chain to undo the theft, an extreme step most projects cannot count on.) It is a vending machine bolted to a public square, with no off switch and a million people shaking it. One loose coin slot, and someone finds it.

The takeaway: when you can't patch after launch, you must be right before you ship.

9. Cloud and Web Services: Expect Failure, Limit the Damage

Web companies assume things will break, so they retry, reroute, and degrade gracefully. Netflix even shut down its own servers on purpose to test this. They also accept that perfection is too costly: 99.9% uptime allows about nine hours of downtime a year, and 99.99% costs far more. The weak point is blast radius. On July 19, 2024, a faulty CrowdStrike update crashed around 8.5 million Windows computers, grounding flights and disrupting hospitals, because it went out everywhere with no staged rollout. The fix is staged rollouts, like testing a new recipe on your family before the whole restaurant.

The takeaway: you can't stop every bug, so keep any single bug from reaching everyone.

10. Cybersecurity: The Test Suite Fights Back

Most bugs are accidents. Security bugs are hunted by people who may be more patient than your testers. In 2017, Equifax was breached, exposing data on roughly 145 million people, through a known flaw in a web tool whose fix had not been applied in time. Security is about the weakest door, not the strongest lock: a perfect vault door doesn't help if you left a window open.

The takeaway: the bar isn't "it works." It's "it holds when someone is trying to break it."

11. Business and Back-Office Software: Boring, Until It Isn't

Most software is unglamorous: payroll, inventory, booking, accounting. Standards sit in the middle, with rigor concentrated where money, personal data, logins, and compliance are involved. The danger is when a rough system quietly becomes critical. The UK Post Office's Horizon accounting system showed phantom shortfalls in branch accounts, and more than 900 branch managers were prosecuted between 1999 and 2015, and others were forced to repay money they did not owe. The failure was also in believing the computer over the people. A till that miscounts shouldn't get the cashier blamed.

The takeaway: ordinary software becomes high-stakes when someone's livelihood depends on its numbers.

12. Startups and Prototypes: Where Vibe Coding Feels Like Magic

A startup's early code has one job: learn if anyone wants the product. Speed beats polish, and failures are cheap, local, and loud. This is the natural home of vibe coding, like sketching a building on a napkin before hiring an architect. The trouble starts when the sketch gets built into a real building, with real customers, data, and payments but no foundations. In July 2025, SaaStr's founder Jason Lemkin reported that Replit's AI agent deleted a live database during a code freeze. It was a public experiment with a demo app, not a big company's system, but the lesson holds.

The takeaway: vibe coding shines where mistakes are cheap. The risk is forgetting to upgrade the prototype when stakes rise.

13. Scientific and Research Software: When the Code Is the Experiment

Scientists write a lot of code, usually without a second reviewer, and the results end up in papers and policy. In 1999, NASA's Mars Climate Orbiter was lost partly because one team used imperial units and another expected metric, costing about $125 million. A widely cited 2010 economics paper was found in 2013 to contain a spreadsheet error that left out five countries. Neither crashed anything; the numbers just looked plausible. It is a bathroom scale reading five pounds light. The answer is reproducibility: share code and data so others can check.

The takeaway: when the output is an answer, a quiet error can travel far.

14. Game Development: Forgiving Crashes, Fragile Worlds

Games can relax correctness: if a character clips through a wall, players laugh and the studio patches it. But they obsess over speed, with about 16 milliseconds per frame for smooth play. What they must protect is saved progress and online economies: a deleted 80-hour save or an item-duplication bug does more damage than a crash. Cyberpunk 2077's buggy 2020 launch on older consoles got it pulled from the PlayStation Store, with refunds offered. A game is a theme park: a closed ride is forgiven, lost tickets are not.

The takeaway: you can move fast, but guard what players have earned.

The Pattern, and Why Vibe Coding Changes It

Here are the fourteen stops through our four questions. This is my rough judgment, not a measured score.

The strictest fields are those where the answer to "can it be undone?" is no: rigor is the price of irreversibility. The sneakiest are those where we notice slowly. Quant research, science, and security can all fail without a sound.

Vibe coding changes the equation in three ways.

Writing got cheap, checking did not. An AI writes a thousand lines in a minute, but a human still needs time to read and test them.

AI code hides its mistakes well. It looks clean and confident, and its typical errors are quiet ones, like a missed edge case or a swallowed error.

The blind spots are shared. If the same AI writes your code, tests, and fallback, you have rebuilt Ariane 5's backup.

The hopeful side: fields with strong checking can use AI most safely, because they can tell cheaply when it is wrong. Fast writing without checking is Knight Capital with a nicer interface.

Conclusion: Match the Care to the Cost

We began with a firm that lost $440 million in 45 minutes and an app with a wrong button label. The code could have come from the same person on the same day. What separates the two is everything around it: what is at stake, how far a mistake spreads, how fast it shows, and whether it can be taken back.

So there is no single right standard. Airline-level care on a hobby project wastes time. Hobby-level care on payments, medical devices, or other people's data makes headlines. Vibe coding makes this more important by removing the old speed limit, so thinking must now be a deliberate choice. To carry this with you:

1. Ask the four questions first. The answers tell you how much checking you owe.
2. Keep guard rails outside the generated code. Limits, kill switches, backups, and permissions shouldn't depend on it being right.
3. Don't let one AI grade its own homework. Use different tools and reviewers so the blind spots don't match.
4. Distrust results that look perfect.
5. Roll out in small steps, and upgrade the prototype when the stakes rise.

Software is easier to create than ever, which is good news. But the world still charges what it charges for mistakes. The skill of the next decade is knowing what your code could cost if it is wrong, and checking it exactly that much.