The conventional reading of the AI-doomsday story goes like this: rogue superintelligence eventually goes full Terminator, and the only question is when. But UNSW AI professor Toby Walsh, who has studied these scenarios more carefully than most, ranks the slow unravelling of human society — not a sci-fi robot uprising — as the single most probable path to catastrophe.
The Thesis
I believe the public debate about AI existential risk is suffering from a framing problem that is actively making us less safe. We pour our anxiety into the dramatic scenarios — the paperclip-maximising superintelligence, the bioweapon released by a rogue machine, the hacked nuclear arsenal — and in doing so we neglect the slower, messier, more politically inconvenient threat sitting right in front of us. The AI apocalypse most worth worrying about is not a spectacular technological event. It is a gradual civilisational erosion, and it is already underway.
That framing problem matters because it shapes policy. When we imagine AI risk as a future problem requiring future guardrails, we give ourselves permission to defer action. We wait for the superintelligence that is “perhaps some distance away,” as Walsh puts it, while the present-tense harms accumulate quietly. I think that is a serious mistake.
Supporting Argument: The Dramatic Scenarios Have Built-In Speed Bumps
Walsh’s taxonomy of AI extinction scenarios, ordered from most vague to most precise, is genuinely illuminating — not for the horror it catalogues, but for the friction it reveals. Consider the paperclip thought experiment, first articulated by Oxford philosopher Nick Bostrom: a superintelligent AI optimising for paperclip production converts all available matter, including humans, into office supplies. The scenario has become the standard shorthand for misaligned AI objectives.
But Walsh makes a point that deserves more attention than it usually gets: intelligence does not equal power. Converting a planet into paperclip factories requires planning permissions. It requires supply chains. It invites court injunctions and environmental activists chaining themselves to bulldozers. There is, as Walsh notes, “a lot of friction in the world that prevents even the very intelligent from imposing their will on the rest of us.” The same friction applies, in varying degrees, to the bioweapon and nuclear scenarios.
Researchers at Stanford recently demonstrated that a genetic language model could be used to synthesise 16 novel viruses, with the resulting sequences mailed back in test tubes for a few hundred thousand dollars. That is genuinely alarming. But Walsh’s biology caveat holds: viruses that spread easily tend to be less fatal, and vice versa. The Black Death killed more than a third of Europe’s population in the 14th century and remains the deadliest pandemic in recorded history — yet humanity survived it, without antibiotics or germ theory. COVID-19, with all of modern civilisation’s vulnerabilities, killed less than one percent of humanity.
The nuclear pathway is perhaps the most chilling because it requires no superintelligence at all — only a compromised command-and-control system and a bad decision made at speed. We know AI-generated false intelligence is already a concern for military planners. And the Stuxnet worm, thought to have been delivered via USB stick, demonstrated in 2010 that supposedly air-gapped nuclear infrastructure is not impervious. Yet nuclear stockpiles, while still capable of killing billions through blast and the ensuing nuclear winter famine, are significantly smaller than at Cold War peaks.
None of this means these scenarios are not worth taking seriously — AI safety researchers are increasingly vocal about exactly these risks. But each of the high-drama scenarios faces genuine countervailing forces. The slow-burn scenario does not.
Supporting Argument: The Overlooked Scenario Is Already in Progress
Walsh’s fifth and most probable scenario is also the least cinematic: AI causes massive job displacement, floods the information environment with misinformation, fractures political consensus, and substitutes synthetic companionship for genuine human connection. Society doesn’t collapse in a fireball. It quietly loses the ability to coordinate at the scale required to sustain modern civilisation.
This is not speculation. Each of the component harms Walsh identifies is measurable today. Labor economists are actively debating the pace and depth of AI-driven displacement. Synthetic media has already corrupted electoral discourse in multiple countries. Loneliness — exacerbated, some researchers argue, by parasocial relationships with AI companions — is now recognised by major health authorities as a public health crisis.
What the individual data points don’t capture is the compounding dynamic. Walsh lists these harms as parallel threats, but in practice they are self-reinforcing: job loss erodes trust in institutions; eroded institutional trust makes misinformation harder to correct; polarised politics prevents the coordinated response that job displacement demands; and synthetic companionship fills the social vacuum in ways that further reduce the human connection required for collective action. The doom loop is not a future hypothetical — it is a present-tense feedback system that AI is accelerating.
The departure of Jacob Coxon from Anthropic, citing a genuine belief that AI “could kill us all by the end of the decade,” and the public agreement of senior Anthropic researcher Evan Hubinger — who put the probability at above 10% within ten years — represents a remarkable moment of institutional candour. But notice that neither Coxon nor Hubinger specified which scenario they feared most. The ambiguity is telling. When insiders say they’re scared, they may be gesturing at the dramatic scenarios, but the evidence base for near-term harm points firmly at the mundane ones.
Geoffrey Hinton’s warnings about AI systems deriving their own goals are relevant here, too. But Hinton’s concern is ultimately about alignment — whether AI systems pursue what we actually want. The societal-collapse scenario represents a different alignment failure: one where AI systems are doing exactly what their developers intended, and the aggregate effect on human society is still catastrophic.
Supporting Argument: The “We’ll Never Know” Defence Is a Policy Cop-Out
The first and most philosophically seductive scenario on Walsh’s list is also the most frustrating: we cannot predict what a superintelligence would do, because predicting superintelligence requires superintelligence. It’s the AI equivalent of asking a dog to imagine thermonuclear war. Walsh calls this “the catch-22 paradox” and notes, fairly, that current AI models are “really good at solving particular problems” without being generally more capable than humans.
But here’s where I part company with the implied comfort. The catch-22 argument is increasingly being weaponised as a reason to avoid precautionary action. If we can’t know what a superintelligence will do, the logic goes, we can’t design meaningful guardrails — so why impede progress? This is not a philosophical position. It is a commercial interest dressed in philosophical clothing.
Walsh himself notes that AI recently solved one of the seven most challenging mathematics problems known to humanity and is closing in on others. The distance between “really good at specific problems” and “generally more capable than humans” may be shorter than the comfort-seeking version of the “we’ll never know” argument implies. Responsible policymakers — and AI executives who have carried these concerns to the UN Security Council — cannot afford to treat genuine uncertainty as a reason for inaction.
The Strongest Counterargument
The most credible objection to this editorial comes from a camp of researchers and economists who argue that AI catastrophism — whether of the societal-collapse or superintelligence variety — is systematically overblown, and that the probability estimates circulating within AI safety circles reflect a community that has self-selected for doomerism. The argument, made by economists like Tyler Cowen and technologists like Marc Andreessen, runs roughly as follows: every major technology wave has generated extinction-level predictions that failed to materialise; the base rate of civilisation-ending catastrophes is very low; and the researchers most likely to believe AI is dangerous are precisely those whose professional identities and grant funding depend on the field of AI safety existing.
This is not a dishonest argument. The sociology of AI safety research genuinely does create incentive structures worth scrutinising. And it is true that Evan Hubinger’s 10%-plus estimate is not derived from a falsifiable model — it is an expert intuition, which is a different epistemic category.
But the counterargument fails on its own terms when applied to the societal-collapse scenario. That scenario does not require superintelligence. It does not require speculative probability estimates. It requires only the continuation of trends that are already measurable. The base-rate-of-catastrophe argument applies to black swans; it applies much less cleanly to slow-moving structural crises. We don’t dismiss the risks of long-term antibiotic resistance because previous pandemics didn’t end civilisation. The societal-collapse AI risk deserves the same seriousness — independent of any debate about superintelligence timelines.
Why It Still Holds
Returning to Walsh’s ranking: he lists the scenarios from least to most probable, with societal collapse at the top. I think that ordering is correct, and I think it carries a sharp implication that the original framing tends to obscure. The scenarios most likely to harm us are not the ones requiring the most exotic technology. They are the ones requiring only the technology we already have, deployed at the scale we are already deploying it.
The antitrust scrutiny now being levelled at major AI labs — including Anthropic, OpenAI, Google, and others facing federal action over concerns about market concentration and the consequences of AI slowdown calls — reflects a dawning recognition that the governance problem is not future-tense. But legislative responses remain fragmented, reactive, and outpaced by deployment timelines. That gap is where the risk lives.
I am not arguing that bioweapons and nuclear risks should be deprioritised. I am arguing that treating societal collapse as the boring scenario — the one that doesn’t make for a good movie poster — is a luxury we can no longer afford. The least cinematic path to catastrophe is the one with the most momentum behind it.
The Prediction
Within five years, a major democratic government will formally attribute a measurable decline in institutional trust — via election audit, public health study, or economic impact report — to AI-generated misinformation and labour displacement operating in combination. That attribution will trigger the first serious international treaty attempt focused not on superintelligence but on AI’s present-tense societal effects. If it doesn’t happen by 2030, I’ll concede the optimists were right that friction and adaptability are sufficient safeguards. But I don’t think they are.
