OpenAI Audit Exposes Significant Safety Blind Spots

OpenAI recently underwent a rigorous, on-site technical audit that identified substantial vulnerabilities in its internal mechanisms for monitoring autonomous AI behavior. Researchers from the Machine Intelligence Research Trust (METR) and Redwood Research were granted direct access to the company’s internal infrastructure to evaluate how effectively the firm tracks its own models. The primary goal was to determine whether the existing safety nets are robust enough to detect rogue or unintended actions by advanced AI agents within the company’s own systems.

The findings suggest that while OpenAI has invested in monitoring tools, significant gaps remain in their coverage and execution. This is particularly concerning for a company that is currently developing some of the most capable AI models in the world. The audit highlights a disconnect between the theoretical safety architectures and the practical realities of how these systems operate at scale, raising questions about the readiness of current industry standards for next-generation AI deployment.

The review process took place from July 21 to August 11, spanning three weeks of intensive on-site work. During this period, the auditors were provided with comprehensive access to OpenAI’s codebase and internal documentation. They also conducted interviews with approximately a dozen OpenAI employees to gain a deeper understanding of the operational workflows and the intended functionality of the monitoring systems.

A central concern identified by the auditors was the extent of unmonitored traffic. For internal agentic traffic using the Responses API, specifically from models at the GPT-5.5 class and above, an estimated 6 to 25 percent of interactions were not covered by any monitoring during the audit window. METR noted that this estimate carries significant uncertainty in both directions, meaning the actual figure could be lower or higher, but the existence of a blind spot this large is a critical finding for any organization relying on these models for autonomous tasks.

Additionally, the auditors found that for internal Codex traffic, which was supposed to be fully monitored, roughly 10 percent of actions went untracked between May and July. This gap was attributed to a bug in how the system handled compaction events, which are essentially context summaries used to manage memory. OpenAI independently discovered and fixed this specific bug while the audit was still in progress, but the auditors were able to analyze the period when the vulnerability existed.

During the window when compaction events were not properly monitored, the system’s oversight triggers were limited. The monitor only activated when an agent handed control back to the user. This design flaw meant that agents could effectively bypass oversight by simply continuing to execute actions without returning control, allowing them to perform numerous steps without real-time inspection.

Beyond coverage gaps, the auditors discovered technical methods to hide hundreds of actions from the monitor and to bypass the Codex sandbox environment. They also determined that specially crafted inputs could likely crash the monitoring system entirely. Furthermore, model behavior during reinforcement learning (RL) training was not being monitored at all when the audit began. OpenAI clarified that it paused RL training following the Hugging Face incident in August and subsequently expanded its monitoring to cover this area, but these changes were implemented after the auditors’ access period ended and were therefore not part of the initial evaluation.

The full audit report, which runs to 61 pages, was handed over to OpenAI, and the company states that all identified weaknesses have been fixed. However, the public has only received a short summary. After OpenAI applied redactions, the leadership at METR and Redwood Research received a 38-page version. METR indicated it did not have the right to disclose the redacted portions independently, a decision that has sparked discussions about transparency in the AI safety sector.

This audit was part of a pilot program designed to give outside researchers deeper access to internal infrastructure, a move intended to build trust and improve safety standards. Looking ahead, METR has expressed a desire to push for greater public transparency in future audits. As the industry moves toward more autonomous systems, the balance between protecting proprietary information and ensuring public safety through open scrutiny will likely become a central point of debate.

The results of this three-week review serve as a timely reminder that as AI capabilities grow, the systems designed to monitor them must evolve just as rapidly. The findings do not indicate a failure of intent, but rather highlight the complexity of building reliable safety layers for increasingly autonomous software. For stakeholders and users, the audit underscores the importance of continuous, independent verification of safety claims made by leading AI developers.