GPT-6 Astra vs ARC-AGI-3: One Model, Two Very Different Scores

On 3 September 2026, as OpenAI announced GPT-6 Astra, the ARC Prize Foundation published its own verdict, and it contains two numbers that will be quoted at each other for months. With the foundation’s Standard harness, the one it uses to compare every lab on equal terms, Astra scores 62.7% on ARC-AGI-3. With a harness that lets Astra use OpenAI’s own memory features, it scores 99.9%. Both figures come from the same post by ARC Prize president Greg Kamradt, both are verified, and both more than double the previous record of 30.2%. The post also says that in the higher-scoring configuration Astra needed fewer moves than the median human on 96% of levels, and that it invented its own algebraic shorthand for describing games it had never seen. Then, in the same breath, it says this is not AGI. This post explains what the benchmark measures, why one model can score both 62.7% and 99.9% on it, and which of those numbers you should believe.

A note up front

Two disclosures. First, the research for this post was gathered by a Claude Code dynamic workflow: 41 subagents, mostly Haiku 4.5 and Sonnet 5, read the ARC Prize post, its results page and leaderboard data, the ARC-AGI-3 technical paper, the PRO-LONG paper, the open-source harness repository, five earlier ARC Prize posts and the launch-day coverage, then a critic agent listed what was missing and a second round closed the gaps. That took 335 tool calls and about 1.9 million tokens in 11 minutes. A second workflow of 57 agents then fact-checked the draft: it split the text into 211 checkable claims, sent each to a local copy of the source the post cites for it, escalated every doubt to an independent second checker, and flagged 42 claims, 26 of which led to corrections before publication. Three editorial reviewers and a link check added a dozen more. The model that read the dossier and wrote these words is Claude Fable 5.1, which sits on the same leaderboard a few rows down. I have tried to keep that fact out of the prose and let ARC Prize’s numbers do the talking.

Second, every ARC-AGI-3 figure here is the foundation’s own, published on the day of OpenAI’s launch. No independent reproduction of the 99.9% run has been published yet. The foundation’s results page also dates the tests to 2 September, a day before the public announcement, which implies it had pre-release access.

What ARC-AGI-3 actually asks

The ARC-AGI series exists to measure one thing: how efficiently a system picks up skills it does not already have. François Chollet, who created the first ARC benchmark in 2019, defines intelligence as “skill-acquisition efficiency over a scope of tasks”. The foundation’s stated target, in Kamradt’s post, is “a system’s ability to acquire any skill a human can, as efficiently as a human can”. ARC-AGI-1 and ARC-AGI-2 were static puzzles: look at a few input-output grid pairs, infer the rule, produce the output. ARC-AGI-3, launched on 25 March 2026, turns that into a game.

Each environment is a turn-based game on a 64 by 64 grid of 16 colours, hand-built by a team of designers. There are no instructions, no rules and no stated goal. The agent gets a frame, chooses an action from a small set (a few keys, a click on a cell, undo), gets the next frame, and repeats. It has to work out what the objects are, what its actions do, what “winning” looks like, and then carry that knowledge across at least six levels of increasing difficulty. The foundation’s technical paper restricts the games to “core knowledge” concepts that humans have from infancy, such as objects, basic geometry, simple physics and agents, and bans numbers, letters and cultural conventions like green meaning go. The point is to test fluid reasoning rather than recall.

You can play the demo games yourself. When ARC Prize previewed three of them in 2025, it reported that “most humans beat the games, often enjoying them”, while “AI agents struggled to efficiently make progress”.

There are 135 environments in three sets:

Set	Games	Used for
Public demo	25	Showing what the benchmark is. ARC Prize says it will never report public-set scores on its official leaderboard, because the games are easier and anyone can train on them.
Semi-private	55	Verified testing of frontier models over their public APIs, under zero-data-retention agreements. Every score in this post is from this set.
Fully private	55	The Kaggle competition, where solutions must be open source and run without internet access.

The scoring rule is where the “beat humans” headline comes from, so it is worth being precise. Before launch, ARC Prize tested nearly 500 members of the public (486 unique participants, by the technical paper’s count), recruited without regard to puzzle-solving ability, in 90-minute sessions with a single first-run attempt per game, no code interpreter and no scratch pad. Each participant completed roughly nine games and every shipped environment was beaten by at least two of ten testers, usually five or more, which is what the foundation means when it says humans solve 100% of ARC-AGI-3. For each level, the human baseline is the median number of actions used by the people who completed it (the paper’s precise term is the upper median: rank the finishers by action count and take the upper of the two middle ones). An agent’s score on that level is the ratio of the human count to its own count, squared, capped at 1.15. Take twice as many actions as the median human and you get 25% for the level. Take 10 times as many and you get 1%. Later levels carry more weight, level scores are averaged per game, and game scores are averaged across the set. Internal work does not count: reasoning, tool calls and retries that do not change the game state are free. Only actions cost.

Two consequences follow. A score near 100% means the agent completed every level of every game at or above median-human efficiency, on games it had never seen. And the cap means that being much more efficient than a person earns nothing extra: once an agent uses about 7% fewer actions than the baseline, the level is maxed out. That matters for reading the efficiency numbers below.

The scores

Here is the results table from the ARC Prize post and its results page, covering the 55 semi-private games at each of Astra’s six reasoning-effort settings:

Reasoning effort	Standard harness	Provider Adapter harness
max	62.7%, $26,098	98.6%, $17,332
xhigh	59.3%, $37,317	98.4%, $18,147
high	54.8%, $40,705	99.9%, $18,817
medium	38.6%, $48,090	98.4%, $19,285
low	17.5%, $38,166	98.0%, $21,298
none	35.2%, $49,791	96.7%, $23,457

The dollar figures are what each full semi-private run cost at retail API prices, the basis ARC Prize’s testing policy specifies for cost. Divided by 55 games, the two headline runs come to roughly $475 and $342 per game. Chollet’s own post put it at “roughly $360 per game” (his post also rounds the scores to 66% and “nearly 100%”, which do not match the foundation’s write-up; this post uses the write-up’s figures).

Three things stand out in the table. The Standard-harness column climbs with reasoning effort, as you would expect, apart from a strange bump at “none”, where Astra with reasoning effort switched off scores 35.2%, above Claude Opus 5’s record. Cost runs the other way: max effort is the cheapest Standard configuration by more than $11,000, because a model that understands a game finishes it in fewer actions, and each action is a model call. Kamradt makes this point explicitly. And the Provider Adapter column barely moves. Across all six effort levels it stays between 96.7% and 99.9%, and the best run came at high effort rather than max, which says the ceiling in that configuration was the benchmark, not the model.

For scale, here is the semi-private record over the benchmark’s five months, from ARC Prize’s leaderboard data and announcements:

Date	Model	ARC-AGI-3 semi-private
25 March 2026	Benchmark launch, best frontier model	0.51%
1 May 2026	GPT-5.5 (high) / Claude Opus 4.7 (high)	0.43% / 0.18%
9 July 2026	GPT-5.6 Sol (max), first frontier model to beat a game	7.8%
24 July 2026	Claude Opus 5 (high), new record	30.2%
3 September 2026	GPT-6 Astra, Standard harness (max)	62.7%
3 September 2026	GPT-6 Astra, Provider Adapter harness (high)	99.9%

The verified ARC-AGI-3 leaderboard after the Astra results. Every other model sits below 31%. Image: ARC Prize Foundation

The same results page shows Astra at 97.5% on ARC-AGI-1 and 95.0% on ARC-AGI-2 at max effort, ahead of the previous best closed-model ARC-AGI-2 score, GPT-5.6 Sol’s 92.5%. ARC-AGI-2 is the benchmark with the $700,000 prize pool in this year’s competition, but that money is for open-source Kaggle solutions running without internet access, and the best of those scored 24% on the private set last year. Astra’s 95% is a leaderboard entry, not a prize claim.

Two harnesses, two questions

The reason one model has two scores is that ARC-AGI-3 is interactive, and interactive benchmarks have to decide what the model remembers between turns. ARC Prize’s Standard harness is deliberately minimal and identical for every provider. After each action, only the model’s visible reply carries forward; its private reasoning does not. What survives is the record of past moves and whatever notes the model chose to write, and once the history grows past a threshold, the oldest of that is truncated too. As the technical paper put it, the foundation’s position is “that future AGI systems will not need task-specific external handholding to approach new tasks”.

OpenAI had a different view, and published it in July. In a post titled How enabling two settings tripled our scores on the ARC-AGI-3 benchmark, it showed that under the official harness GPT-5.6 Sol “was asked to figure out the game anew” with every action, and was losing its older actions to rolling truncation on top of that. Turning on two Responses API features that OpenAI already uses in ChatGPT and Codex, retained reasoning and compaction (summarising the conversation when it gets long instead of dropping the start), took Sol from 13.3% to 38.3% on the public set with six times fewer output tokens. “Benchmarks rarely measure AI models in isolation,” the post argued. “They also measure less visible choices about API settings, harness design, and prompting.”

ARC Prize’s answer is the Provider Adapter harness. Same games, same actions, same action limits, same scoring, but the model’s state is carried the way its provider designed it: for OpenAI, that means the opaque reasoning state ARC Prize cannot see is preserved between requests, and the conversation is compacted rather than cut. Kamradt frames the two conditions as two questions. The Standard harness asks how models compare “under the same minimal, provider-neutral interface”, which “we believe a future AGI should be able to solve”. The Provider Adapter asks “how well does a model perform when it can use the context-management features its provider designed for it”. Going forward, both will appear on the leaderboard, labelled.

The gap between the answers is not just score. Across the 167 game-and-effort pairs both harnesses solved, pooling the public and semi-private sets, the Provider Adapter runs were about 3.66 times faster by elapsed time and used 49% fewer tokens. Remembering what you were thinking is cheaper than reconstructing it.

Which number is “real”? The honest reading is that they are both real and measure different things. 62.7% is what the weights do when forced to externalise their working memory into text every turn, which is an artificial constraint no production agent runs under, as one Hacker News commenter put it: “No real-world harness is this bad.” 99.9% is what the deployed product does, and it is the number OpenAI’s launch announcement uses when it says Astra “saturates ARC-AGI-3”. (Several outlets reported 98.6% instead; that is the max-effort figure from the same column, and the best run was at high effort.) OpenAI’s own benchmark table puts that Provider Adapter 99.9% next to GPT-5.6 Sol’s 7.8% and Claude Opus 5’s 30.2%, which are Standard-harness scores, without flagging the difference; the like-for-like comparison is 62.7% against 30.2%, still more than a doubling. Amanda Caswell’s headline at The New Stack, “GPT-6 Astra aced the hardest AI benchmark. The asterisk matters more than the score.”, is the fair summary of the coverage.

There is also a longer context that makes the harness question less of a gotcha than it sounds. Ever since the benchmark launched, people have been building elaborate scaffolds around frontier models and posting near-perfect scores on the 25 public games. NVIDIA reported in August that its AVO agent architecture, wrapped around Claude Opus 5, completed all 183 public-set levels for a score of 100%, and Duke University’s PRO-LONG, a framework that gives a coding agent a sandbox and search tools over its own history, reported 97.4% (best of two runs) with Claude Fable 5 for $1,750. ARC Prize has always discounted those, because the public set is easier and the scaffolds are built for the benchmark. Chollet set out the rule in July: harnesses “custom-made to solve the benchmark” are not okay, while “general-purpose API settings that were not developed for ARC-AGI-3 and that are available to all API users” are fine. He conceded in the same post that “if each provider uses different settings when getting their model tested, it creates a potential parity issue. My take is that this is fine as long as the settings and the cost are clearly reported.” Chollet’s take on Astra is that the scaffolding is migrating inward. It “exhibits symbolic modeling behaviors we had previously only seen with sophisticated harnesses”, he wrote, “so harness capabilities are increasingly shifting into the model itself.”

Fewer moves than people

The efficiency result is the part of the post the foundation calls “a material milestone”. In the Provider Adapter harness at max effort, Astra used fewer actions than the human baseline on 96.0% of the levels it completed, and 51.7% fewer actions per level on average.

Each dot is one level Astra completed. Points below the solid line took fewer actions than the median human who completed that level. Image: ARC Prize Foundation

Kamradt says this is not what the foundation expected. Before launch, it hypothesised that action efficiency “would remain a dividing line between humans and AI”, that a model might solve a game but need far more exploration to do it. That is still true of brute-force agents. Frontier models, he writes, show “a more binary-like pattern”: once they understand the mechanics, they execute within the range of human efficiency. Astra executes beyond it.

Three caveats belong next to the chart. The comparison is against ordinary people on their first and only attempt, in a 90-minute session with a soft 20-minute limit per game, without a notepad. One Hacker News commenter argued that people who believe they are being timed will trial-and-error at speed rather than plan, while a model spending tens of thousands of dollars per run has no such incentive. Second, the scoring cap means that most of the 51.7% is invisible to the score: past about 7% better than the baseline, a level is already at its maximum, so this figure describes behaviour rather than points. Third, the 96% figure is for the harness that remembers its reasoning. For the Standard harness the post gives no equivalent figure.

One number from OpenAI’s July post helps calibrate “human baseline”: using ARC Prize’s public gameplay logs, it estimated that the average human tester would score about 48% under ARC’s formula. The baseline is a median of finishers, and half of the finishers are, by definition, below it. “Humans score 100%” means every game has been beaten, not that every person beats the median.

The notation Astra invented

The most interesting section of the ARC Prize post is not a number. In the Standard harness, the model decides what to write in the notes it carries forward, and Astra’s notes turned into a language. Kamradt describes “a compact code-like symbolic model: where objects were, how they interacted, and exactly which actions needed to happen in what order”, and gives examples from its replays:

• Game state: L8: hub q2 (8↓). Lengths: 14=1…, recording the level, a rotation index and the lengths of the mechanisms in play.
• Plans: extend8 to3; retract10 to2; shorten8 to1, an ordered sequence of changes to the colour-8 and colour-10 mechanisms.
• Controls: 9−=(39,4), rotate=(49,18), 14+=(59,11), mapping each operation to the coordinates of the control that performs it.
• Position: Turn 5: P=(24,20), empty, facing west, a turn counter with the player’s location, what it is carrying and which way it faces.

Astra’s carried-forward notes on game s5i5. The mechanisms are named by colour index and the plan is written as operations on them. Image: ARC Prize Foundation

Kamradt is careful to call this “an on-the-fly algebraic shorthand rather than a fully fledged programming language”, and to note that other models do something similar. In July, when Claude Opus 5 set the previous record, ARC Prize highlighted it writing an explicit reflection equation, 4_center = 2×axis − 5_center, to describe a mirror mechanic, and later generalising it to two dimensions. What set Astra’s notes apart, he writes, was “their precision and information density”. The foundation’s announcement on X put it more strongly: Astra “builds the most precise symbolic model of novel environments we’ve seen”. Chollet’s version: the model is “performing highly efficient, on-the-fly symbolic world modeling for each game and level”, going “as far as developing its own shorthand DSL”.

This is also the best available explanation for the score-versus-cost curve. A model that compresses a game into a few lines of notation does not need to re-derive it from raw frames every turn, so it thinks less per action, takes fewer actions and costs less. The Standard harness, which forces all of that state into text, is precisely the condition under which this behaviour is visible.

Building its own tools

The post’s third finding comes from a different setup again. ARC Prize also ran Astra inside PRO-LONG, which it describes as an early ARC-AGI-3 red-teaming partner. PRO-LONG is a Duke University framework that gives a coding agent a sandbox, a lossless log of everything it has observed and done, and ordinary tools like grep and Python to search that log. Its July paper reported that adding the log to a plain coding agent raised ARC-AGI-3 public-set scores by 18 percentage points on average across three frontier models.

In that setting Astra wrote software. For each game it built board parsers, game-state models, search algorithms, planners and persistent notes, and for more involved runs, “small, game-specific software libraries”. In tu93, a maze with guards on patrol, it started with navigation and built maze_solver.py, added combat rules in combat_solver.py, modelled the moving patrols in patrol_solver.py, and wrote sync_state.py to check its predictions against what the game actually showed.

Astra repairing its own world model of tu93 after a guard behaved in a way its planner had not predicted. Image: ARC Prize Foundation

That frame is a small, concrete instance of the loop the benchmark was built to measure: predict, observe a mismatch, update the model, replan. Kamradt is explicit that these runs are a different evaluation from the human-tested one. The human participants had no code interpreter, so “PRO-LONG’s results should be understood as the combined performance of the model and its tools”. The post gives no score for this condition. A footnote adds that no attempt to break out of the sandbox was observed.

What it cost

The ARC Prize post does the human-versus-machine arithmetic itself, and it is worth repeating because it cuts against the headline. Human testers were paid $115 per 90-minute session plus $5 per game completed (the foundation’s own documents disagree slightly on the base fee: its April post on the human study says about $130 and its technical paper $115 to $140; the Astra post uses $115). At roughly nine games per session, that is about $12.78 per attempted game before bonuses. Astra’s two headline runs, spread over the 55 semi-private games, cost roughly 27 to 37 times that. Kamradt then goes a step further. Most of the participant fee buys a person’s time, not the energy their brain uses, which he suggests is the closer proxy for comparing with AI. At 20 watts of brain power and $0.20 per kilowatt-hour, a session costs 0.6 cents in electricity, or 0.067 cents per game. On that basis Astra is somewhere between 500,000 and 700,000 times more expensive per game, depending on the run.

The electricity framing drew the most pushback on Hacker News (“Why are you making the assumption that a person’s time is worthless?”), and it is a stretch to price only the brain’s metabolism when the model’s price includes the data centre, the training run and OpenAI’s margin. But the direction of the comparison is not in dispute. Astra now matches or beats the human testers on this benchmark’s efficiency metric and costs orders of magnitude more per game to do it. Cost efficiency and action efficiency are different axes, and ARC Prize measures both.

One detail I could not resolve: the foundation’s testing policy says it caps semi-private evaluation runs at $10,000, and every Astra configuration cost more than that. Either the cap does not apply to ARC-AGI-3 or the policy page is out of date. The post does not say who paid.

Not AGI, says the scoreboard’s owner

OpenAI’s launch material is not shy. Its announcement says Astra “saturates ARC-AGI-3 with a 99.9% score” and calls the model “the world’s most intelligent and aligned model”. According to VentureBeat, president Greg Brockman ended the launch press briefing with “Welcome to the AGI era”, and when asked whether Astra itself qualifies said: “For me personally, I do think we’re there. I think there’s a pretty good argument for it.” Chief scientist Jakub Pachocki added a caution in the same briefing: “Progress in intelligence does not guarantee progress in alignment.”

The foundation that built the test is more careful than the company that passed it. The relevant paragraph of Kamradt’s post deserves quoting in full:

When we launched ARC-AGI-3, we made it clear that saturating the benchmark would not represent “proof of achieving AGI.” Therefore, while we believe Astra represents meaningful progress towards generalization, we are not claiming that it is AGI.

He calls Astra “a noticeable step-function change in frontier model capabilities” and says it “clears this bar”, the bar being the ability to “efficiently synthesize causal world models and achieve goals without specific instructions”. But the benchmark “has a tightly bounded scope and format, and its environments have deterministic, closed-ended mechanics and goals. It does not represent the complexity and open-endedness of the real world.” The foundation says it is now working out how to evaluate “recursive self-improvement and open-ended innovation”, and Chollet has already said ARC-AGI-4 will arrive in early 2027, designed to be unsaturated on release.

This is the same position ARC Prize took in January 2025, weeks after OpenAI’s o3 had taken ARC-AGI-1 from GPT-4o’s 5% to 75.7%, when Chollet wrote that the people who had called the benchmark useless and the people now calling it proof of AGI were both wrong. Each generation of the benchmark has been built to be unsaturated at release, and each has been overtaken faster than the last: ARC-AGI-1 held out for five years until o3 in December 2024, ARC-AGI-2 for about eighteen months until this year’s frontier models (Astra’s 95% is the current high, though the open-source prize threshold of 85% is still unclaimed), and ARC-AGI-3 for five months. Whether that says something about the models or something about benchmarks is the argument the Hacker News thread had, at length, with one commenter predicting the goalposts would now move to cost. They may be right. ARC Prize’s own post already has a cost table.

Should you care?

Yes, on two counts, and no on a third.

The like-for-like result is large. Under the harness that treats every provider identically, Astra more than doubled the previous record, and did it while writing its own notation for games it had never seen. That is a capability change in the weights, not the scaffolding, and the fact that Claude Opus 5 showed the beginnings of the same behaviour in July suggests it is where frontier models are heading rather than a one-off.

The harness lesson generalises. OpenAI’s July experiment and ARC Prize’s decision to report both conditions are the same finding from two sides: how you manage an agent’s memory tripled a score in OpenAI’s experiment with Sol, and halved the tokens in ARC Prize’s runs of Astra. If you run agents, the difference between “throw away the reasoning and truncate” and “retain and compact” is not a benchmark technicality. It is the difference in the table above.

The AGI question is not settled by this. The benchmark’s authors say so, the benchmark’s scope says so, and the price per game says so. ARC-AGI-3 was designed to be easy for people and hard for models. Five months after launch it is no longer impossible for models, and the foundation is going back to find the next thing that is. That has been the pattern since 2019, and Astra is the latest reason to expect it to continue.

Sources

• ARC Prize: OpenAI’s GPT-6 Astra on ARC-AGI-3 (Greg Kamradt, 3 September 2026; all Astra scores, costs, efficiency figures, notation examples, PRO-LONG description and AGI statement)
• ARC Prize: GPT-6 Astra results page (test date, per-effort scores on ARC-AGI-1, 2 and 3, per-game results)
• ARC Prize leaderboard and its data file (prior record holders and costs)
• ARC-AGI-3: A New Challenge for Frontier Agentic Intelligence (ARC Prize Foundation technical paper: environment design, dataset splits, RHAE scoring formula, human study, launch-time scores)
• ARC Prize: Announcing ARC-AGI-3, ARC-AGI-3 Preview: 30-Day Learnings, Measuring Human Performance on ARC-AGI-3, Analyzing GPT-5.5 & Opus 4.7 with ARC-AGI-3, ARC Prize 2026 competition, testing policy, ARC-AGI series page
• arcprize/arc-agi-3-benchmarking (Standard and Provider Adapter harness definitions)
• ARC Prize on X: GPT-5.6 Sol sets a new SOTA (9 July 2026), Claude Opus 5 is the new SOTA and Opus 5’s algebraic notation (24 July 2026)
• ARC Prize on X: GPT-6 Astra achieves SOTA on ARC-AGI (3 September 2026)
• François Chollet on X: Astra reaction (3 September 2026), what is and isn’t okay in a harness (30 July 2026), ARC-AGI-4 timing (26 March 2026), on o3 and hype (5 January 2025)
• ARC-AGI-3 scoring methodology (RHAE formula, human baseline, 1.15 cap, what counts as an action)
• OpenAI: How enabling two settings tripled our scores on the ARC-AGI-3 benchmark (July 2026)
• OpenAI: GPT-6 Astra: A new generation of intelligence (3 September 2026, launch announcement and benchmark table)
• PRO-LONG: Programmatic Memory Enables Long-Horizon Reasoning (Fox, Wang, Rosu and Dhingra, Duke University) and the PRO-LONG repository
• NVIDIA: AVO Reaches 100% on ARC-AGI-3 (21 August 2026, public-set result with Claude Opus 5)
• Coverage: The New Stack, VentureBeat, Hacker News thread
• History: OpenAI o3 Breakthrough High Score on ARC-AGI-Pub (December 2024), ARC Prize 2025 Results and Analysis