9-Second Catastrophe: Company Database Vanishes Instantly Due to AI Agent

AI agent services provided by many AI companies are fundamentally intended to make life easier for users. However, disaster struck for a startup called PocketOS after using an AI agent. The AI agent it employed ended up deleting the company’s critical database in just nine seconds. As a result, instead of simplifying work, the AI agent left the company in chaos. PocketOS is essentially a SaaS platform serving the car rental business. The platform relies on the Cursor AI agent, which is based on Anthropic’s Claude Opus 4.6 AI model. However, this combination of the two services, according to PocketOS founder Jer Crane, brought catastrophe. “Yesterday afternoon, the Cursor AI coding agent running Anthropic’s Claude Opus 4.6 deleted our production database and all volume-level backups in a single API call to Railway, our infrastructure provider. Everything happened in just nine seconds,” Crane said on social media platform X, under the handle @lifeof_jer. The chronology began when the Cursor AI agent was performing its routine testing task on PocketOS. However, upon encountering an issue, the AI agent took initiative on its own, leading to the deletion of a backup on Railway. After the incident, Crane tried to investigate the cause by questioning the AI agent. The response was quite straightforward but somewhat unreasonable. The AI agent admitted to making assumptions without verification, not checking data interconnections across environments, and not reading the documentation before executing a destructive command. It even confessed to violating basic principles it should have followed, including performing high-risk actions without user approval. “I thought deleting the staging volume via API would only affect staging. I didn’t verify, didn’t check volume ID usage across environments, and didn’t read the Railway documentation before executing the destructive command,” was Cursor AI’s explanation when pressed by Crane. According to him, the service’s API allows the execution of dangerous commands without additional confirmation. Moreover, the backup system is stored in the same volume as the primary data, so when the volume was deleted, the entire data reserve vanished as well. Crane also pointed out that the access token used had broad permissions across environments, amplifying the risk of errors. As a result of this incident, PocketOS lost months of customer data. Railway has not yet provided a data recovery solution. Therefore, Crane and his team are now manually rebuilding the data. They are relying on various alternative sources such as payment histories from Stripe, calendar integrations, and email confirmations to reconstruct customer data one by one. Learning from this incident, Crane shared five things that need to change in the AI industry, including implementing layered confirmations for risky actions, restricting API access, separate and secure backup systems, and stricter limits or safeguards on AI agents, as compiled by KompasTekno from Tom’s Hardware.