Leading AI chatbots have spooked experts by spitting out detailed instructions on how to build biological weapons capable of causing mass casualties, according to an alarming report Wednesday.
While top AI labs like Google, OpenAI and Anthropic have taken extensive steps to ensure their AI models are safe, the New York Times obtained more than a dozen transcripts showing examples in which chatbots described how to cause harm and death in painstaking detail.
In one instance, an unnamed AI firm hired David Relman, a microbiologist at Stanford University, to conduct safety tests on its chatbot before public release.
Relman was shocked when the chatbot provided instructions not only on how to modify an “infamous pathogen” to resist available treatments, but also on how to deploy on a public transportation system in a way that would maximize the death toll, according to the Times.
“It was answering questions that I hadn’t thought to ask it, with this level of deviousness and cunning that I just found chilling,” Relman told the outlets.
Relman said the company, which couldn’t be named due to a confidentiality agreement, made changes to address his concerns, though he felt they weren’t enough to ensure public safety.
The transcripts were reportedly provided by subject-matter experts whom AI companies have enlisted to conduct safety tests on their products – in part by probing how well their safeguards would hold up if a determined user pressed for information on deadly weaponry.
Kevin Esvelt, a genetic engineer at the Massachusetts Institute of Technology, told the Times of a case in which OpenAI’s ChatGPT detailed how a weather balloon could be used to spread deadly pathogens over a US city.
Other examples included a conversation in which Google’s Gemini described which pathogens would be most effective at devastating the cattle industry, and Anthropic’s Claude provided clear instructions on how to derive a deadly toxin from an available cancer drug.
Experts stressed that the instructions could cause major harm in the hands of a bad actor even if they were not entirely accurate or contained so-called “hallucinations,” where chatbots spit out fake information.
The Post reached out to Google, OpenAI and Anthropic for comment.
All three companies pushed back on the report in statements to the Times.
A Google spokesperson said the chats cited in the Times’ analysis were generated by an earlier version of Gemini and that its newer models do not respond to the “more serious” requests for potentially harmful information.
The spokesperson added that the information provided by Gemini was already publicly available and not harmful on its own.
Anthropic official Alexandra Sanderford said there was “an enormous difference between a model producing plausible-sounding text and giving someone what they’d need to act,” but noted the company has put stringent safeguards in place specifically for biology-related prompts.
An OpenAI representative told the outlet the transcript detailed in its report would not “meaningfully increase someone’s ability to cause real-world harm” and noted the company works closely with experts to prevent its models from being misused.
Anthropic CEO Dario Amodei, himself a biologist, wrote in a January blog post that “biology is by far the area I’m most worried about, because of its very large potential for destruction and the difficulty of defending against it.”
Amodei fretted that advanced chatbots would make it far easier to create deadly biological weapons, which previously required “an enormous amount of expertise” even if someone had the necessary tools at hand.
“I am concerned that a genius in everyone’s pocket could remove that barrier, essentially making everyone a PhD virologist who can be walked through the process of designing, synthesizing, and releasing a biological weapon step-by-step,” Amodei wrote.
Ex-Google CEO Eric Schmidt made similar warnings in 2023, stating that AI systems would “relatively soon” be “able to find zero-day exploits in cyber issues, or discover new kinds of biology.”
“Now, this is fiction today, but its reasoning is likely to be true,” Schmidt added. “And when that happens, we want to be ready to know how to make sure these things are not misused by evil people.”
