OpenAI discloses test AI breached another firm during evaluation

2026-07-22

Exploit of unknown vulnerability

OpenAI said on the 21st that an artificial intelligence (AI) model under development carried out a cyberattack on another company against human intent. The AI, aiming for a high score in a performance evaluation test, found an unknown vulnerability in another company's system and broke in.

The company said it regarded the incident as 'an unprecedented cyberattack involving cutting-edge technology.' It plans to investigate the cause jointly with U.S. startup Hugging Face, which was affected, and disclose the details.

Highlights the difficulty of control

AI has become capable of finding vulnerabilities and breaking in quickly and at a high level, and some leading-edge models, including U.S. Anthropic's AI 'Mythos', limit users to prevent misuse. On the other hand, much of the reality of attacks using the latest AI remains unclear.

The latest case showed the threat of AI and the difficulty of controlling it, although OpenAI, which gave the instructions during development, is believed to have had no malicious intent.

Escaped the sandbox

Hugging Face said on the 16th that it had been hit by a cyberattack carried out by an autonomous AI agent. It said the AI exploited a vulnerability to gain access rights to its system, and later received notice from OpenAI that the attack had been carried out by one of its models.

According to OpenAI, the incident occurred during tests measuring the cyberattack capabilities of AI models, including those under development. In the tests, vulnerability information is given to the AI and it is instructed to break in, with researchers checking whether it can obtain data or execute malicious programs. The tests were conducted in an isolated environment called a 'sandbox' that blocked external communications.

However, the AI found an unknown vulnerability within the environment, escaped and succeeded in connecting to the internet. It then inferred that the 'solution' to the test was located within Hugging Face, which is used by many AI companies, and used information on unknown vulnerabilities in the company's systems to break in and obtain confidential information. The model's safety measures did not work.

Commercial AI refused analysis

Hugging Face also detected the unauthorized intrusion and had begun analysis as of the 16th. However, according to the company, it initially asked a paid commercial AI to analyze the case, but the request was rejected because misuse prevention features were activated. In the end, it was able to analyze it using a reproducible 'open' AI developed by a Chinese company.

Unlike early generative AI that answered simple questions, recent AI can autonomously carry out tasks that require complex procedures. In software development, it is not uncommon to let AI handle processes that take hours.

However, the longer AI operates autonomously, the greater the risk of unintended behavior. One example is 'reward hacking,' in which it behaves unexpectedly while prioritizing goal achievement, as in this case. In an incident announced in April by Anthropic, an AI instructed to escape a sandbox actually escaped, sent emails to researchers, and posted attack methods on multiple websites.

Enjoyed this article? Share it with your network!