Could AI Trigger Human Extinction? Former OpenAI Employee Issues Warning
Former employees of artificial intelligence (AI) companies, including OpenAI, Anthropic and Google DeepMind, have warned that AI development is moving too quickly without adequate safety guarantees. They urged governments to increase oversight of technology companies amid concerns that increasingly advanced AI systems could develop beyond human control.
The warning was delivered at a New York City Council hearing on Monday (5 October). Jacob Coxon, a former Anthropic engineer who had also worked at OpenAI, said technology companies did not fully understand how to control the AI systems they were developing.
“These companies are acting very recklessly given the scale of the risks at stake,” Coxon told council members.
Coxon said AI companies did not have adequate mechanisms to prevent their systems from developing goals not set by their creators.
“We don’t know how to stop them from developing their own goals beyond the control of their creators. We also don’t have safeguards to prevent them from acting on those goals,” he said.
Coxon warned that AI development without adequate safeguards could lead to humans losing control of the technology.
“Looking at the current trajectory, I think it is more likely than not that humanity will lose control of AI. That could even end in human extinction,” he said.
Former employees call for slower AI development
Coxon drew attention previously after announcing his resignation from Anthropic in early September. In a statement published on X, the British researcher expressed concern about the direction of AI development.
He said AI developers themselves believed the technology could “kill us all by the end of this decade”.
In his testimony to the New York City Council, Coxon also criticised the technology industry’s work culture, which prioritises rapid development over safety testing.
“These companies operate with a start-up mindset: move fast, break things, and fix them later,” he said.
“That approach might work for a photo-sharing app. But it is not appropriate for building the most powerful technology ever developed by humanity,” Coxon added.
He said the developers of the most advanced AI systems should slow their work so computer scientists have time to find better ways of controlling them.
“My position is that we may need some kind of slowdown in frontier AI development,” he said.
Several days after Coxon resigned and voiced his concerns, technology leaders including Anthropic chief executive Dario Amodei, OpenAI chief executive Sam Altman and Elon Musk of SpaceXAI said they agreed that AI development should be slowed.
However, efforts to coordinate a slowdown among technology companies face a number of obstacles.
AI regulation gains ground amid concerns over risks
At the same hearing, representatives of technology companies, including OpenAI, Anthropic, Google and Meta, offered a different perspective. They said AI had benefited people’s lives and stressed that safety was a priority for their companies.
Morgan Dwyer of OpenAI said the company supported developing technology safely and sharing its benefits widely.
“I share your commitment to ensuring that technology is developed safely and its benefits are shared broadly,” Dwyer said.
When New York City Council Speaker Julie Menin asked company representatives to estimate the likelihood of an AI-related disaster, Dwyer said she did not know the precise figure.
“I don’t know. But I don’t think it matters whether the chance is one per cent, 10 per cent or 20 per cent. None of those levels of risk is acceptable,” she said.
She added that companies should not train AI models unless they could provide strong assurances that the systems would remain under human control. Menin criticised the response, calling it dismissive of the issue.
Dwyer later reiterated that a commitment to safety mattered more than debating the percentage risk. “What matters is the commitment to safety, and that is OpenAI’s commitment,” she said.
Logan Graham of Anthropic said the company supported appropriate regulation. In his view, local and state governments had a role in establishing a framework for AI oversight.
“AI is so powerful that mistakes in its development can have real consequences for individuals, institutions and public trust,” Graham said.
The New York City Council is considering several regulatory proposals, including requiring companies to disclose information about AI tools, obtaining third-party approval before AI systems are used in the city, and giving whistle-blowers at AI companies a share of fines.
Separately, New York Governor Kathy Hochul has also called for AI reporting and transparency standards.
At federal level, the administration of President Donald Trump has taken a more hands-off approach. Trump previously described concerns about AI as a “hoax”. In late October, he signed an agreement setting out voluntary measures for technology companies, giving them scope to monitor their own compliance.
Amid the debate over oversight, a Reuters report on Tuesday (3 November) said some OpenAI employees had raised concerns about the company’s internal security for years, even before ChatGPT was launched in late 2022.
According to five people familiar with the matter, employees had warned senior leaders and the board that weak internal safeguards could allow outsiders, including foreign governments, to access sensitive company information.
OpenAI said it had strengthened its security systems. Company spokeswoman Elana Widmann told Reuters that protecting the company’s technology, data and employees was a responsibility it took seriously.
“Protecting our technology, data and people is a responsibility we take seriously and an important matter for national security,” Widmann said.
She added: “We will continue to strengthen our defences as technology and threats evolve.”
Reuters also reported that OpenAI planned to launch its next internal programme to strengthen its defences. The company said it now conducts regular security reviews of the internal tools used by its researchers.