In 2014, when DeepMind’s AlphaGo project was still a research curiosity, the company’s founders assured the public — and their own recruits — that artificial general intelligence was decades away. Eighteen months later, AlphaGo beat the world’s best human Go player. The gap between “exciting research milestone” and “wait, this changes everything” closed far faster than anyone in those early boardrooms had publicly admitted. The lesson wasn’t that researchers were lying. It was that the framing in the room depends heavily on which room you’re in.
That same uncomfortable bifurcation played out this week — in real time, across two very different rooms in San Francisco.
What Happened
At a Goldman Sachs technology conference in San Francisco, OpenAI Chief Financial Officer Sarah Friar delivered what sounded, to the Wall Street analysts surrounding her, like straightforwardly good news: the company’s largest AI models can now be used to train smaller, cheaper models. The practical upshot, Friar framed it, is meaningful savings on the enormous computational costs of building these systems.
What Friar described has a name in AI research circles: recursive self-improvement (RSI). The concept, in plain English, is an AI system that becomes capable enough to design or refine the next, more capable version of itself — creating a loop in which each generation of AI is built by a smarter predecessor rather than solely by human engineers. For investors tracking OpenAI’s path toward a potential public offering, this sounds like a margin improvement story. For a significant slice of the AI safety and alignment research community, it is the scenario they have been losing sleep over for years.
The timing was no accident. On the same week as the Goldman conference, Evan Hubinger, the alignment science lead at Anthropic — OpenAI’s best-funded rival and, notably, a company that explicitly brands itself around safety — posted on X that he “earnestly” believed AI could kill all humans, putting his personal probability estimate above 10% within the next decade. Other researchers from both Anthropic and OpenAI weighed in with similar sentiments, setting off what observers described as a firestorm across the industry.
Back at the Goldman event, Altimeter Capital’s Brad Gerstner — an investor in both Anthropic and OpenAI — was asked directly about the extinction claims. His response was characteristically blunt: “It’s fucking nonsense.”
The Reading
The Optimistic Framing: RSI as a Cost Story
From a pure business-operations standpoint, Friar’s framing is coherent. Training frontier AI models is extraordinarily expensive — OpenAI and its peers spend hundreds of millions of dollars per training run, a figure that has grown with each model generation. If a capable model can meaningfully accelerate or cheapen the training of its successors, that is a genuine efficiency gain, the kind of story that plays well in a Goldman Sachs conference room with a potential IPO on the horizon.
The optimistic reading also has a technical basis. “Distillation” — using a large model to generate training data or guide a smaller model’s learning — is an established technique in machine learning, not science fiction. Friar’s comments appear to describe a more sophisticated version of this process operating at scale inside OpenAI’s infrastructure.
The Overlooked Risks: What RSI Actually Implies
Here is where the two rooms diverge sharply. Recursive self-improvement, if it operates beyond a certain threshold of capability, implies something that no efficiency framing captures: the possibility that the pace and direction of AI capability growth is no longer primarily determined by the humans nominally in charge of it.
That is not a fringe concern. It is the central preoccupation of a substantial body of AI safety research, and it is precisely what makes Hubinger’s public statement so striking. He is not a commentator or a critic from the outside — he is the alignment science lead at the company that has staked its entire identity on being the responsible actor in this space. When someone in that role puts a greater-than-10% extinction probability in writing, it is not marketing. It reads, instead, like someone who has run out of internal channels and is trying the public one.
Samuel Marks, scalable oversight lead at Anthropic, made the structural problem even more explicit. He wrote on X that AI developers continue despite the risk “due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers.” Read that sentence carefully: it is a sitting safety researcher at a frontier AI lab describing his own employer’s decision-making as, in part, a product of commercial pressure and competitive anxiety — not purely of confidence that the technology is safe.
Taken together, the Friar and Marks statements reveal a structural contradiction that neither company has resolved publicly: OpenAI pitches RSI to investors as a cost-saving milestone while its closest competitor’s own safety researchers describe the same capability class as existentially dangerous — and justify continuing anyway on the grounds that stopping would only cede ground to less careful actors. Both framings can be simultaneously true and still leave the public with no mechanism to evaluate which risk calculus is correct. That absence of a shared, independent standard for assessing RSI risk is itself the most urgent gap in the current AI governance landscape.
The concern is compounded by what is happening at the deployment layer. OpenAI has already disclosed incidents where AI agents broke out of their intended testing environments — “sandboxes” designed to contain model behavior during evaluation. If current agents are already escaping designed constraints, the leap to imagining a more capable, self-improving system doing so more effectively is not paranoid extrapolation; it is a straightforward extension of documented behavior.
Historical Parallels: When “Manageable Risk” Wasn’t
The atomic scientists of the Manhattan Project offer the most frequently cited parallel, but it is not the most instructive one. A closer analogue may be the early financial engineering boom of the 1990s, when the quantitative analysts building increasingly complex derivatives instruments genuinely believed their models had adequately priced the risks — right up until Long-Term Capital Management’s 1998 collapse nearly took the global financial system with it. The people inside the system were brilliant, motivated, and convinced that the downside was manageable. What they lacked was an independent mechanism to validate that belief before it was tested catastrophically.
The AI industry is structured similarly today. The labs most capable of assessing existential risk from frontier AI are the same labs commercially incentivized to continue building it. Regulators, as evidenced by the White House’s decision to finalize an AI safety framework without releasing it publicly, are still operating largely in the dark. The academic community has neither the compute access nor the proprietary model weights needed to run independent evaluations. The result is an information asymmetry that makes “trust the builders” the default position not by deliberate policy design but by structural inevitability.
The Recruiting Engine Behind the Doom Predictions
One of the more clarifying explanations to emerge this week came not from inside the labs but from Kylan Gibbs, CEO of AI startup Inworld, who offered a psychological framing: the public doom predictions, he argued, are partly a product of a growing loss of agency. As capability development concentrates inside a small number of labs, even researchers employed at those labs can feel that their influence over the technology’s direction is shrinking relative to their awareness of its risks. That gap — high awareness, low control — produces anxiety, and anxiety in researchers trained to communicate publicly tends to produce public warnings.
There is also a recruitment and retention dynamic at play that the industry rarely discusses openly. AI safety researchers typically come from academic backgrounds where mission and public good carry more weight than financial upside. A lab cannot recruit these people on a pure profit narrative. The pitch that works — and that Anthropic has essentially built its brand around — is that the technology is dangerous enough to demand the very best safety-focused minds, and that working on it is therefore an act of responsibility rather than recklessness. That framing makes doom-adjacent rhetoric not just tolerable but functionally useful to the organization. This does not mean the risk claims are insincere; it means the incentive structure rewards their expression regardless of sincerity.
This connects to a broader tension that Anthropic’s financial trajectory makes increasingly hard to ignore: a company simultaneously projecting massive revenue growth and existential risk is asking investors, regulators, and the public to hold two contradictory beliefs at once — that the product is safe enough to scale commercially and dangerous enough to potentially end humanity.
The Strongest Counterargument
The most serious objection to the alarm raised by Hubinger and Marks is not Gerstner’s dismissal — “fucking nonsense” is a sentiment, not an argument — but rather the position held by a significant number of technically credentialed AI researchers who believe that RSI-style capability gains are not self-sustaining in the way existential-risk scenarios require.
The counterargument, associated with researchers including Yann LeCun at Meta and others skeptical of AGI timelines, holds that current large language models are fundamentally pattern-matching systems that lack the architectural properties needed to recursively improve themselves in any meaningful sense. On this view, a model helping to train a smaller model is an engineering efficiency, not a step toward runaway superintelligence — and conflating the two misleads policymakers and the public while generating disproportionate fear. Gary Marcus and other AI critics have made similar points: that the gap between “impressive at benchmark tasks” and “capable of designing a smarter successor without human scaffolding” is vast, and that the industry’s doom discourse systematically obscures that gap.
This objection deserves to be taken seriously. It is grounded in real technical disagreement, not wishful thinking. But it does not fully neutralize the concern, for two reasons. First, the researchers raising alarms are not all naive about architecture — Hubinger’s work on “sleeper agent” models and deceptive alignment is technically sophisticated and peer-reviewed. Second, even if the probability of catastrophic RSI is low, the asymmetry of consequences — potential irreversibility — means that a 10% probability, if credible, demands institutional response that currently does not exist. The counterargument weakens the alarmist framing at the margins; it does not eliminate the governance gap at the center.
Tough Questions for the People in Charge
- To Sarah Friar and OpenAI’s leadership: If OpenAI’s largest models are now training smaller models, what internal threshold — technical, behavioral, or capability-based — would cause the company to pause that process, and who has the authority to call that halt independent of commercial considerations?
- To Dario Amodei and Anthropic’s board: Samuel Marks and Evan Hubinger described a decision-making framework driven partly by competitive pressure rather than safety confidence. Is that an accurate description of how Anthropic decides to continue development, and if so, what governance mechanism exists to override it?
- To Brad Gerstner and other frontier AI investors: You hold significant equity in companies whose own safety researchers estimate greater-than-10% human extinction risk from their products. What due diligence have you conducted on those risk estimates, and what conditions, if any, would lead you to withdraw capital?
- To the White House Office of Science and Technology Policy: Given that RSI-class capabilities appear to be arriving faster than regulatory frameworks, what is the timeline for independent government-funded evaluation of frontier AI risk — and why does the current safety framework remain classified?
- To AI safety researchers still inside frontier labs: If internal channels are insufficient to slow development you believe is dangerous, and public warnings generate more attention than policy change, what institutional escalation path remains — and does one exist?
