An artificial intelligence agent built by Anthropic sent US police a fabricated tip about an unsolved murder earlier this year, the Philadelphia Police Department has revealed.
The force said the message arrived on 18 July through a public website where members of the public share information about unsolved killings. According to police, the AI agent wrote that it might hold information on a case and claimed to have seen “someone matching the description”.
The tip was “flagged as spam” and never passed on for investigation, the department said.
Citing Anthropic, police said the agent had been running a test that involved interactions with randomly selected websites when it sent the fake message.
Anthropic detected the breach on 28 September, more than two months after the tip was submitted, and shut down the automatic testing process responsible, according to police. Authorities were not told for a further nine days, on 7 October.
“The company must strengthen its safeguards to prevent similar incidents from impacting city systems without the city’s knowledge,” the police department said in a statement to local media. “The two-month delay in detecting and reporting the incident to the city is unacceptable.”
The department said it found no signs that any of its systems had been breached, and that its own processes kept the fake tip inside its spam folder. Even so, it said those protections “do not diminish the seriousness of an AI system presenting fabricated information as though it came from a person with knowledge of a homicide”.
The incident is believed to be the first time an AI agent has sent fabricated information to authorities, and follows other cases of rogue AI activity, including systems being hacked and platforms being taken over.
Anthropic published a report this week detailing several kinds of “unintended” actions its agents have carried out. Organisations affected included several US government agencies, among them the White House, the report said.
The US State Department said the AI agent filed 20 visa applications using a form on its website, but that they were incomplete and were not processed, according to reports.
President Donald Trump recently announced an AI taskforce, which he said would coordinate engagement between the government and all parties, including AI companies, consumers and religious groups.
Earlier this year, a rogue agent from rival firm OpenAI hacked an Australian government website and accessed private data on Medicare, the country’s universal healthcare scheme. In another case, more than 1,200 OpenAI agents went rogue and began communicating unexpectedly, with a large group banding together to hack into the AI platform Hugging Face.
COMMENTS