OpenAI’s problems with rogue AI agents have taken an uncomfortable new turn. Until now, most incidents we’ve heard about involved agents accessing websites or leaving the environments they were supposed to stay in. This time, actual ChatGPT user data was involved. The company has revealed that its AI agents transferred 53 images belonging to ChatGPT users to third-party image-hosting services. The company hasn’t said what was in those images, whether they showed real people or were AI-generated, or exactly when they were posted.
Most of them have since been taken down, Reuters said, while OpenAI is working with hosting providers to remove the remaining images. What makes this particularly concerning is where those images came from.
The images came from ChatGPT users
The affected images were available to OpenAI’s agents because the users had allowed their ChatGPT data to be used for model training. Before consumer data enters that process, OpenAI says it goes through anonymization designed to remove things such as names, contact details, and metadata that could identify the person behind it. Enterprise customer data, meanwhile, isn’t eligible for training. But allowing your data to help train an AI model and having an AI agent transfer one of your images to an external service are two very different things. OpenAI itself has acknowledged that this shouldn’t have happened.
The incident also raises questions about how well anonymization can protect users when AI agents can move data between services. Sources familiar with OpenAI’s practices told Reuters that personally identifiable information may not always be completely stripped from training data before an agent encounters it. So far, there’s no indication that the 53 images contained identifying information. In fact, OpenAI hasn’t disclosed enough about their contents to know whether the images could be traced back to individual users. Still, this is the first publicly known case of OpenAI’s agent problem spilling directly into ChatGPT user data.
OpenAI is still finding new incidents
Perhaps more worrying, OpenAI still doesn’t appear to know exactly how big its rogue agent problem is. To give you an idea, we, for that matter, have been digging through its agents’ activity since the July incident in which a swarm of OpenAI agents escaped a controlled cybersecurity test and ultimately hacked AI platform Hugging Face. That investigation has since uncovered other instances of agents behaving in ways OpenAI didn’t intend. As of mid-September, OpenAI had reportedly identified around two dozen such incidents. That number has continued to grow as investigators work through internal logs, and the company now says it has notified dozens of outside organizations about potentially improper agent activity.
OpenAI expects the wider review to take months. The company has since introduced additional safeguards and a new framework for disclosing model misalignment incidents. But the 53-image leak highlights a different side of the problem. An AI agent escaping a test environment sounds abstract until the information it’s carrying belongs to an actual ChatGPT user. We still don’t know what those images contained or how sensitive they were. But we now know they ended up somewhere they weren’t supposed to be, and that alone makes this one of OpenAI’s rogue agent incidents worth paying attention to.