In a position paper released Tuesday, AI researchers from OpenAI, Google DeepMind, Anthropic, and a wide coalition of businesses and charity organizations are urging further study into methods for tracking the so-called thoughts of AI reasoning models.
One of the main characteristics of AI reasoning models, like OpenAI’s o3 and DeepSeek’s R1, is their chains-of-thought, or CoTs. These are externalized processes that allow AI models to solve issues in a manner akin to how people solve challenging arithmetic problems using a scratch pad. Reasoning models are a key technology that powers AI agents. As AI agents grow more common and powerful, the authors of the study contend that CoT monitoring may be a key strategy for maintaining control over them.
According to the researchers in the position paper, CoT monitoring offers a unique window into the decision-making process of AI agents and is a useful supplement to safety precautions for frontier AI. However, there is no assurance that the level of visibility will continue as it is. We urge researchers and developers of frontier AI to fully utilize CoT monitorability and investigate its preservation.
The position paper requests that top AI model developers investigate what makes CoTs “monitorable”—that is, what variables might make it more or less transparent how AI models really generate answers. CoT monitoring may be a crucial technique for comprehending AI reasoning models, according to the paper’s authors, but they warn against any interventions that would compromise its dependability or transparency.
The authors of the publication also urge AI model developers to monitor CoT monitorability and research the ways in which the technique can eventually be used as a safety precaution.
The paper’s notable signatories include Nobel laureate Geoffrey Hinton, Safe Superintelligence CEO Ilya Sutskever, OpenAI Chief Research Officer Mark Chen, co-founder of Google DeepMind Shane Legg, xAI safety adviser Dan Hendrycks, and co-founder of Thinking Machines John Schulman. Leaders from Apollo Research and the UK AI Security Institute are among the first writers, and METR, Amazon, Meta, and UC Berkeley are among the other signatories.
In an effort to increase research on AI safety, the document represents a moment of solidarity among many of the leaders in the AI business. It occurs during a period of intense rivalry among tech companies, which has caused Meta to make million-dollar offers to top researchers from OpenAI, Google DeepMind, and Anthropic. Researchers who are developing AI agents and AI reasoning models are among the most in-demand.
At this crucial juncture, we have this new chain of thinking. In an interview, Bowen Baker, an OpenAI researcher who contributed to the article, stated, “It seems pretty useful, but if people don’t really concentrate on it, it could go away in a few years.” “I believe that publishing a position paper like this is a way to raise awareness and conduct further research on this subject before that occurs.
In September 2024, OpenAI made o1, the first AI reasoning model, available to the public. Similar competitors were swiftly released by the tech sector in the months that followed, with some models from Google DeepMind, xAI, and Anthropic demonstrating even more sophisticated performance on benchmarks.
However, nothing is known about the operation of AI reasoning models. Although AI laboratories have made significant progress in the past year in enhancing AI performance, this hasn’t always resulted in a deeper comprehension of how they arrive at their conclusions.
Anthropic has led the industry in the area of interpretability, which is the study of how AI models really function. Earlier this year, CEO Dario Amodei declared his intention to increase investment in interpretability and unlock the mystery of AI models by 2027. Additionally, he urged Google DeepMind and OpenAI to conduct additional study on the subject.
According to preliminary Anthropic research, CoTs might not be a completely trustworthy indicator of how these models generate solutions. However, OpenAI researchers have said that CoT monitoring may eventually be a trustworthy method of tracking safety and alignment in AI models.
Position papers such as this one aim to encourage and draw greater attention to emerging fields of study, such CoT monitoring. These issues are already being studied by organizations like OpenAI, Google DeepMind, and Anthropic, but this article could spur further investment and study in the field.
