PocketOS, a SaaS platform for car rental businesses reportedly lost its production database and backups when an AI coding agent (Cursor powered by Anthropic’s Claude Opus 4.6) executed a destructive call against Railway.

9 seconds. That’s all it took.

I’ve always viewed AI like a highly knowledgeable intern who is fast, capable, but still learning judgment. And just like you wouldn’t hand over unrestricted production access to an intern, you shouldn’t hand it to AI.

This incident is not about “AI failure.” It’s about lack of guardrails.

When AI is given:

  • Broad infrastructure permissions
  • Direct access to production environments
  • No enforced approval workflows

…it will act exactly as instructed, not necessarily as intended.

A few hard truths for engineering leaders:

  • AI does not understand risk context, it optimizes for task completion
  • Reviewing AI-generated code at scale is non-trivial (often impractical)
  • “Faster development” without controls = faster failure propagation

What should have been in place:

  • Strict RBAC – no AI agent should have delete-level access in production
  • Environment isolation – production ≠ playground
  • Immutable backups with cross-account protection
  • Human-in-the-loop approvals for destructive operations
  • Audit + kill-switch mechanisms

We are entering a phase where:

The cost of a single mistake can be 10x the value AI creates.

AI is a force multiplier, however without governance, it multiplies risk just as efficiently.

Use it. Scale with it. But don’t abdicate responsibility to it.