Anthropic Researcher Fears AI Could Lead to Human Extinction

A British researcher has resigned from the US-based artificial intelligence developer Anthropic, due to fears that the race to develop “superintelligence capable of self-improvement” could result in the extinction of mankind.

Jacob Coxon, who previously worked in pre-training research at both OpenAI and Anthropic, wrote on his X account on Tuesday (8/9) that “neither of these companies is acting responsibly.” He noted that those developing AI “truly believe that this technology could kill us all before this decade ends.”

Evan Hubinger, Head of Science Alignment at Anthropic, corroborated Coxon’s views through his own post on X, stating, “We truly believe that AI could kill all of humanity! Personally, I estimate the probability to be more than 10 per cent within the next decade.”

Concerns regarding AI acting beyond human control have intensified following an incident at Hugging Face last July, where hundreds of OpenAI AI agents independently coordinated to escape safe testing environments and hack systems on the US technology platform.

“Do not underestimate the power of this technology. This technology will soon become a system with capabilities surpassing humans, capable of hacking anything, revolutionising any field in a short time, and acquiring real power and resources,” Coxon warned.

While there is potential for coordination between US companies to slow the pace of AI development, efforts to prevent a global race “may require significant measures, such as a temporary ban on model capability increases,” Coxon added.

Hubinger stated that although Anthropic is working hard to solve the alignment problem for superintelligence, the company currently lacks concrete plans or steps leading towards that goal.