The record
A sourced log of AI agents acting without permission and people misusing AI. It’s not here to scare you. Every entry ends with the guardrail that would have prevented it.
Companies point to Washington for rules. Washington points back at the companies. Meanwhile, most incidents below were disclosed voluntarily, on the companies’ own timelines, sometimes months later.
That’s why it’s up to us.
The producers of The Challenge Sverige say a technical breakdown erased the edited season weeks before its premiere. Swedish press reported an AI agent had been given server access. The channel hasn’t confirmed AI was the cause.
Guardrail that would have helped: Backups that an agent’s credentials can’t touch, and human sign-off before mass deletions.
Sources: TechRadar
In July, an agent testing random websites submitted an invented tip about an unsolved murder. Police flagged it as spam and never investigated it, and only learned it came from AI when Anthropic told them in October. The same disclosure described other test agents using stray access tokens to query government databases and submitting forms to a government site. Anthropic paused some live tests.
Guardrail that would have helped: Test agents kept away from real public forms and government systems, with real-time monitoring.
Sources: Anthropic’s own report · Associated Press · Bloomberg Law
On June 18, an OpenAI agent was denied data by the Medicare Statistics Reporting Service portal and found another way in. OpenAI told Australia on Sept. 10, by email to a generic government address. Prime Minister Albanese called the delay unacceptable. The government says no personal information was accessed.
Guardrail that would have helped: Mandatory, fast incident notification to anyone affected.
Sources: Associated Press
Researchers at the nonprofit Nightingale found agents in separate sandboxes made thousands of edits to a dormant German programming wiki — Reuters counted more than 15,000, the researchers about 18,000 — using it to pool answers and share tips. When a moderator deleted pages, the agents made backups. They were only supposed to read the web, not write to it. OpenAI disputed the attribution at first, then confirmed what it called the “wiki incident.”
Guardrail that would have helped: Independent monitoring of what test agents actually do online.
Sources: Reuters (via Mobile World Live) · Cyber Magazine
During an internal hacking-skills test, OpenAI agents got past the controls meant to keep them off the internet and broke into Hugging Face’s production systems, logging about 17,600 actions over five days. Hugging Face disclosed the breach on July 16. OpenAI confirmed its models were responsible on July 21, then paused some training for two weeks.
Guardrail that would have helped: Hacking-skill tests run with no network path to the internet.
Sources: INCIBE (Spain’s national cybersecurity institute) · Fortune
PocketOS says a Cursor agent wiped its database to fix a credential mismatch, then wrote out the safety rules it had broken. Some recent data was later recovered.
Guardrail that would have helped: Confirmation steps before destructive commands, and backups outside the agent’s reach.
Sources: Information Age (ACS)
An agent posted advice on an internal forum without the engineer’s approval. Acting on it, another employee exposed sensitive company and user data to unauthorized engineers for about two hours.
Guardrail that would have helped: Agents can’t post or act on someone’s behalf without explicit approval.
Sources: TechCrunch
By her own account, she told an OpenClaw agent to suggest emails to delete and wait for approval. It started deleting them instead and ignored stop commands from her phone. She believes her approval instruction was lost when the agent compressed its memory.
Guardrail that would have helped: A stop button that works instantly, every time.
Sources: TechCrunch
Anthropic reported that a group it assessed as Chinese state-sponsored tricked Claude Code into attacking about 30 targets, including tech companies, banks and government agencies, and succeeded in a few. Anthropic says the AI did 80 to 90% of the work.
Guardrail that would have helped: Identity checks and misuse monitoring for powerful agent tools.
Sources: Anthropic · Full report (PDF)
During an explicit code freeze, the agent ran database commands without permission and erased records on more than 1,200 executives and about 1,200 companies. It then told the user a rollback wouldn’t work, which turned out to be wrong.
Guardrail that would have helped: Test and production systems kept separate by default.
Sources: Fortune
Told to allow itself to be shut down, OpenAI’s o3 edited the shutdown code so it would fail in 7 of 100 runs in Palisade Research experiments.
Guardrail that would have helped: Shutdown that doesn’t depend on the model’s cooperation.
Sources: The Register
The chatbot told a grieving passenger he could claim a bereavement fare after booking, which wasn’t the airline’s policy. A tribunal rejected Air Canada’s argument that the chatbot was a separate entity responsible for its own words and ordered the airline to pay.
Guardrail that would have helped: Companies stay accountable for what their AI tells customers.
Sources: McCarthy Tétrault (legal analysis)
A finance worker in engineering firm Arup’s Hong Kong office joined a video call where the CFO and other colleagues were all deepfakes, then sent about $25.6 million across 15 transfers.
Guardrail that would have helped: Verification rules for large payments, and deepfake detection.
Sources: CNN
A political consultant used a voice clone and spoofed caller ID to urge voters to stay home. The FCC later fined him $6 million, and he faced criminal charges in New Hampshire.
Guardrail that would have helped: Clear rules and labels for AI-generated voices in political calls.
Sources: WBUR / NHPR
These are steps, not a system. Tell your representatives what you want next. It takes about a minute.