ARC-AGI is a set of coloured-grid puzzles where each task follows a rule you have never seen, demonstrated by two or three example pairs, and you must apply it to a new input. It was designed so that scaling training data does not help — and the two mechanisms that make that design stick are a genuinely private test set and a published compute budget.
What ARC-AGI is
The Abstraction and Reasoning Corpus was introduced by François Chollet in the 2019 paper On the Measure of Intelligence. The paper’s argument is that a test of intelligence should measure skill acquisition efficiency — how well a system handles a task it has not been prepared for — rather than skill at tasks it was prepared for. Most benchmarks measure the second thing. ARC was built to measure the first.
Concretely, each task is a handful of input-output grid pairs. Grids are up to 30 by 30 cells, each cell one of ten colours. The transformation from input to output is a rule — gravity, symmetry completion, counting, flood fill, object selection, some composition of those — and you infer it from the demonstrations alone. There is no natural-language description of the rule anywhere.
The rules are drawn from what the paper calls Core Knowledge priors: objectness and cohesion, basic geometry and topology, numbers and counting, and goal-directedness. These are the things a young child has and a search engine does not, which is the point — you cannot look the answer up because the task is unique.
The shape of one task
Task (illustrative, not from the corpus). Grids as digit matrices,
0 = background.
Demonstration 1 Demonstration 2
input output input output
0 0 0 0 0 0 0 3 0 0 0 0
0 5 0 -> 0 0 0 0 3 0 -> 0 0 0
0 0 0 0 5 0 0 0 0 0 3 3
Test input Your answer
0 0 7 ? (the rule must be inferred from
0 0 0 the demonstrations alone — no
0 0 0 description is provided)
Scoring is exact match on the full output grid: correct size,
correct colour in every cell. One cell wrong scores zero.
The example above is illustrative rather than a real corpus item, and real tasks are considerably subtler. What it shows correctly is the interface: a few pairs, no instructions, exact-match grading.
Four sets, and why the private one exists
| Set | Description |
|---|---|
| public training | A few hundred tasks, deliberately easier, published in full. Intended for developing an approach, not for reporting. |
| public evaluation | A few hundred tasks at the intended difficulty, published in full. Any score on this set is a score on data that has been on the public internet for years. |
| semi-private evaluation | Held back, used to score commercial systems that must be called over an API. Exposed to the API provider by necessity, which is why it is 'semi'. |
| private evaluation | Never published, never sent to a third-party API. Used for the leaderboard's verified scores, run inside a sealed environment with a fixed compute budget. |
The private set is the mechanism. Every other benchmark on this cluster publishes its test items, and from the moment it does, contamination exposure only increases — the argument in does. It is effectively pass@2 with a hard cap.
The second half of the score is the budget. Verified leaderboard entries are reported with a cost per task alongside the accuracy, because an unbounded search over programs can solve grid puzzles given enough compute, and a benchmark about efficiency of skill acquisition that ignored compute would be measuring the wrong thing. This is one of the few places where a public leaderboard treats cost as a first-class axis rather than a footnote — see should state its extraction rule.
Version details and set sizes here are described at the level worth relying on. The exact task counts per set have changed between releases and the prize rules change annually. Check the ARC Prize organisers’ own current documentation before quoting a specific figure.
What a score does and does not license
A verified private-set score, reported with its compute budget, is one of the more meaningful numbers in public evaluation. It says: on tasks this system had no opportunity to memorise, under a stated budget, it solved this fraction. Very few benchmarks can make the first half of that claim honestly.
What it does not say is anything about language, knowledge, code, tools, or any task with a natural-language interface. ARC-AGI is a deliberately narrow instrument. A system could score highly on it and be useless at everything you would actually deploy a model for, and a system could be excellent at your work and score poorly here.
It also does not settle the question its name invites. The organisers are explicit that saturating the benchmark would not constitute general intelligence — it would mean this particular resistance to
SOCIAL SHARE CARD GENERATOR