AI labs are continuing to uncover novel ways to optimize performance on the ARC-AGI benchmarks, which are designed to challenge frontier artificial intelligence models with complex, abstract logic puzzles. OpenAI has published a new technical analysis showing that its GPT-5.6 Sol model nearly tripled its score on the demanding ARC-AGI-3 public task set, rising from an initial 13.3% to 38.3%. This optimization allowed OpenAI’s model to surpass Anthropic’s Claude Opus 5, which previously held a benchmark record of 30.2%.
However, the dramatic score increase was not achieved through the benchmark’s default testing environment. Instead, OpenAI ran the model using its proprietary Responses API with two specific settings enabled: “Retained Reasoning” and “Compaction.” The development has reignited an industry-wide debate over whether benchmark scores reflect a model’s intrinsic general intelligence or the engineering of the test harness surrounding it.
The Core Problem: Why the Default Harness Throttled GPT-5.6 Sol
According to OpenAI researchers Ilan Bigio and Ted Sanders, the investigation began after GPT-5.6 Sol registered a low score of 7.8% on early ARC-AGI-3 tests, while the older GPT-5.5 model barely registered at 0.4%. These poor results contrasted sharply with GPT-5.6 Sol’s demonstrated capabilities in other complex environments. The model had already successfully cleared Pokémon FireRed on a vision-only configuration, navigated Slay the Spire 2 using Codex’s computer-use tools, and solved early stages of the abstract puzzle game Baba Is You.
The researchers identified two primary culprits within the standard ARC-AGI-3 testing harness that severely degraded the model’s performance:
- Discarded Reasoning: After every action taken by the AI agent, the harness threw away its private chain-of-thought reasoning. Consequently, the model had to re-analyze the state of the puzzle from scratch on every single turn, despite having access to a basic log of past moves.
- Rolling Truncation: The default harness discarded the oldest parts of the conversation once the context exceeded 175,000 characters. This caused the model to lose track of crucial observations made during earlier phases of a run.
This combination forced the model to repeatedly dwell on individual actions, preventing it from building on top of its established understanding of the game’s rules.
The Technical Fix: Retained Reasoning and Compaction
Because GPT-5.6 Sol was built to interface with OpenAI’s Responses API, the researchers modified the benchmark’s implementation to leverage native platform features. By passing along the previous response ID, the model kept its reasoning persistent across turns by default. Additionally, OpenAI replaced the harness’s rolling truncation with “Compaction”—a feature that summarizes historical context rather than deleting it outright.
With these two adjustments, GPT-5.6 Sol spent significantly less time re-deriving rules before executing moves. The model not only cleared all six levels of the specific puzzle game OpenAI tested—where no frontier model had previously bypassed the first level under the official harness—but it also used roughly six times fewer output tokens per run. OpenAI’s broader takeaway is that benchmark outcomes are highly dependent on harness design, advising developers to utilize the Responses API, retain reasoning, and enable context compaction for long-running tasks.
ARC Prize and François Chollet Respond
The ARC Prize Foundation, which administers the benchmark, acknowledged OpenAI’s impressive technical results while defending the integrity of its standardized testing methodology. The foundation clarified that its official, verified leaderboard scores intentionally utilize a “no harness” setup. Under this protocol, every model receives identical observations, system prompts, and action limits, with the conversation state managed strictly on the client side via a standard completions interface.
ARC Prize co-founder François Chollet responded to the development by drawing a distinction between two types of test environments. He noted that harnesses custom-made to solve a specific benchmark or containing pre-loaded knowledge about the task format remain strictly off-limits. However, Chollet conceded that general-purpose API settings that are available to all developers and not built specifically for ARC-AGI-3 are fair game, acknowledging that the standard test environment had put OpenAI’s model at a disadvantage.
ARC Prize stated its goal is to ensure that progress on the benchmark represents genuine advancements in artificial general intelligence (AGI) rather than lab-specific API formatting. The organization confirmed it is actively collaborating with OpenAI and other AI labs to explore how server-side state management can be integrated into official, verified testing without compromising competitive fairness.