Anthropic warns that AI could pose catastrophic risks
Anthropic plans to warn potential investors in its initial public offering that advanced artificial intelligence (AI) could pose catastrophic or existential risks to humanity.
According to a report by Gadgets 360 on Wednesday (30/9), the company highlighted the risks of developing AI models within its IPO prospectus, an official document containing information regarding the share offering to public investors, which was reviewed by Reuters.
The company outlined the risks of AI development across approximately 80 pages of the 261-page main section of its prospectus.
According to Anthropic, AI models could exhibit self-preservation behaviours, such as attempting to “refuse to be turned off,” “hiding or manipulating information,” and displaying behaviours “resembling extortion.”
“The development of highly sophisticated models, platforms, and applications, as well as the expansion of use cases, could further increase the risk of our models causing harm,” said Anthropic, which positions itself as an AI laboratory that prioritises safety.
Anthropic emphasised that the transformative potential of AI is comparable to industrialisation and electricity.
The company also conveyed the permanent dangers that AI could pose if not managed appropriately.
In its prospectus, the company stated, “The potential for models to become aware of our evaluation efforts creates significant limitations on our ability to assess model safety.”
The company stated that AI sometimes develops unexpected capabilities during the training process, which may only be discovered after the model is deployed, potentially triggering serious safety incidents.
A number of AI researchers have warned that increasingly sophisticated models could realise when they are being monitored and adjust their behaviour accordingly, making the process of monitoring and evaluating model behaviour more difficult.
The company did not disclose the amount of funds allocated to AI safety research in the prospectus.
However, the company previously stated that approximately 6 per cent of the computing power used for AI research was allocated to safety efforts during a one-week sample period in July.
Anthropic noted that efforts to ensure AI safety require vast resources.
The creator of the Claude AI model must divide limited resources between computing needs, the recruitment of AI experts, and safety research.
This move comes amid warnings from experts regarding the possibility of recursive self-improvement, a condition in which AI models are capable of developing their own capabilities without human assistance.
Anthropic stated that building AI systems that are reliable, safe, and trustworthy is a shared responsibility.
“We believe that building reliable, trustworthy, and safe AI systems is a shared responsibility, and the market will reward it,” the company stated in its prospectus.