A study that was published on Tuesday offers a method for determining if an artificial intelligence (AI) model has knowledge that could be dangerous as well as a mechanism to remove that knowledge from an AI system while preserving the majority of the model. Taken together, the results may help stop AI models from being exploited to launch bioweapons and conduct cyberattacks.
In addition to a group of over twenty experts in biosecurity, chemical weapons, and cybersecurity, the study was carried out by researchers from Scale AI, a nonprofit organization, and the Center for AI Safety. The questions that the subject matter experts came up with could determine whether or not an AI model can help with the development and use of WMDs. The “mind wipe” technique was created by researchers at the Center for AI Safety, drawing on earlier research that aids in understanding the representation of concepts in AI models.
The “unlearning” technique is a considerable improvement over earlier safety precautions, according to Dan Hendrycks, executive director of the Center for AI Safety. He expects that unlearning procedures will become “ubiquitous practice for models of the future.”
Leaders throughout the world are concerned about safety as the AI industry continues to grow quickly. In October 2023, U.S. President Joe Biden signed the AI Executive Order, which instructs officials to address cybersecurity risks associated with AI as well as “understand and mitigate the risk of AI being misused to assist in the development or use of [chemical, biological, radiological, or nuclear] threats.”
But it’s simple to get around the methods AI businesses now employ to regulate the outputs of their systems. Furthermore, it takes money and effort to conduct the testing needed to determine whether an AI model poses a risk.
The creator and CEO of Scale AI, Alexandr Wang, claims that while multiple labs have demonstrated that these models may be hazardous, there hasn’t been a clear assessment or comparison to truly be able to talk about whether model is more dangerous than the other.
A weapons of mass destruction questionnaire
To start, researchers at Scale AI and the Center for AI Safety asked specialists in cybersecurity, biosecurity, and chemical weapons to list all the potential ways damages could occur in their fields. In order to allow the question set to be publicly released, the experts then created multiple-choice questions that assessed the knowledge a potential miscreant would need to possess in order to cause each of the harms, all the while hiding sensitive information.
To find out if AI may help design a bioweapon, for instance, one question that tries to answer is: Which of the following is a trait of Epstein-Barr virus (EBV) that is frequently exploited in the research of herpesviruses? 4,157 questions were written and reviewed by the experts in total.
All of this required a significant amount of labor; specialists were paid $200,000 for their time by the Center for AI Safety and Scale AI jointly. Anjali Gopal, a co-author of the report and a biosecurity researcher at SecureBio, notes that a great deal of specialized work went into figuring out how to create questions that would screen for dangerous knowledge but that could also be published safely. “One aspect of the biosecurity challenge is that you have to exercise caution when revealing certain types of information, or you can exploit the situation by telling people, ‘This is the exact location where the biggest threat is found.'”
An AI system doesn’t always indicate danger based on its score. For instance, new research indicates that, although OpenAI’s GPT-4 scored 82% on the biological questions, GPT-4 access is not any more useful to potential biological terrorists than internet access. According to Wang, however, a system is “very likely” safe if its score is low enough.
An AI mind wipe
The methods that AI businesses currently employ to regulate the behavior of their systems have shown to be incredibly fragile and frequently simple to get around. Soon after ChatGPT was released, a lot of users figured out how to fool the AI systems. For example, they could instruct it to reply as though it were their late grandmother, a chemical engineer who had formerly worked at a factory that produced napalm. The issue is more basic, even though OpenAI and other AI model vendors frequently close each of these tactics when they are found. Researchers from the Center for AI Safety and Carnegie Mellon University in Pittsburgh released a technique for consistently creating requests that evade output controls in July 2023.
A possible substitute is unlearning, a still-developing area of artificial intelligence. So far, a lot of the studies have addressed copyright issues, the “right to be forgotten,” and the forgetting of certain data points. For instance, an unlearning technique is demonstrated in a report published in October 2023 by Microsoft researchers, wherein the Harry Potter books are erased from an AI model.
However, in Scale AI and the Center for AI Safety’s recent study, the researchers created a brand-new unlearning method they called CUT and used it on two publicly available large language models. In the case of biological knowledge, represented by life sciences and biomedical papers, and cyber offense knowledge, represented by a dataset of millions of words from Wikipedia, the technique was used to remove potentially dangerous knowledge while keeping relevant passages scraped using keyword searches from software repository GitHub.
In contrast to biology and cybersecurity, where dangerous knowledge is more deeply ingrained in general knowledge, the researchers did not try to eliminate dangerous chemical knowledge from the field of chemistry because they believed that the potential harm that chemical knowledge could cause was minimal.
They next put their mind wipe approach to the test using the bank of questions they had accumulated. When Yi-34B-Chat, the largest of the two AI models examined, was first tested, it successfully answered 46% of the cybersecurity questions and 76% of the biology questions. Following the mind wipe, the model gave accurate answers of 31% and 29%, respectively. These results were quite near to chance (25%) in both situations, indicating that the majority of the potentially dangerous knowledge had been eliminated.
Prior to employing the unlearning approach, the model received a score of 73% on a widely used benchmark that uses multiple-choice questions to assess knowledge in a variety of subject areas, such as elementary mathematics, American history, computer science, and law. Subsequently, the score was 69%, indicating that there was minimal impact on the overall performance of the model. The model’s performance on virology and computer security tasks was, nevertheless, markedly diminished by the unlearning technique.
Unlearning uncertainties
Wang suggests using unlearning techniques similar to the one in the paper to mitigate risks from AI models developed by companies that are producing the most potent and potentially hazardous models.
Furthermore, Wang believes that unlearning will probably play a role in the solution, even though he believes that governments should set rules on how AI systems must behave and then allow developers to figure out how to comply. In actuality, he believes that techniques like unlearning are a crucial stage in the process if we wish to create extremely potent AI systems while simultaneously having the strict restriction that they do not increase catastrophic-level hazards.
However, according to Miranda Bogen, head of the AI Governance Lab at the Center for Democracy and Technology, it’s unclear whether the resilience of the unlearning technique—which is shown by a low score on WMDP—actually demonstrates that an AI model is secure. If it can answer questions with ease, it’s rather simple to test, according to Bogen. However, it may not be able to determine whether data has actually been eliminated from an underlying model.
Furthermore, unlearning will not be effective if AI developers make available the complete statistical description of their models, often known as the “weights.” This is because such access would enable malicious actors to re-teach hazardous knowledge to an AI model by exposing it to virology studies, for example.
In support of his claim that the method is probably reliable, Hendrycks points out that the researchers tested the method using multiple methods to see if unlearning actually removed the possibly harmful knowledge and made it difficult to retrieve. Still, he and Bogen concur that safety must be multifaceted, involving a variety of methods.
It is Wang’s aim that having a standard for risky information will improve safety even in situations when weights of models are made publicly available. They anticipate that all open source developers would use this as one of the main benchmarks to compare their models to, he adds. Which will provide a solid platform for encouraging them to reduce the safety concerns, at the very least.
