Nine models made 324 plans in a game of strategy, each played against 1,349 entries from people. The average model loses to the average person, and bigger models plan better.
Each side spreads 100 units over ten prizes worth 1 to 10, without seeing the other's plan. More units wins a prize, a tie splits it, and more points wins. There is no best plan: a plan is only as good as the plans it meets. Every model plan was played against every valid human entry.
What people did
The median human entry beats 52% of the others, and the top tenth beat 71% or more.
FiveThirtyEight's Riddler, 2017
Average AI planAverage person's plan
The two average plans, rounded to whole units, played against each other. Each prize goes to the side with more units; a tie splits it.
What we found
Six findings. Each model counted once. Open any finding for the detail.
Finding 1 · Not supported, prediction B1
The average AI plan beats 38% of people. The median person beats 52%
No model reaches the median person; GPT 6.1 Sol comes closest at 50%.
More on this finding
No model reaches the median person; GPT 6.1 Sol comes closest at 50%. The best single plans are strong: one by GPT 6.1 Sol beats 77% of people. The models can find good plans, they just do not do it reliably.
What is newBenchmarks say models reason well. Against real people in a strategy game the average model loses, and none beats the median person.
38% of human plans beaten, 95% interval 33% to 43%, against 1,349 human entries
An uneven spread avoids predictable proportional bidding while concentrating enough visits to contest several valuable shops.
Finding 2 · Half supported, prediction B3
Agents fall back on a plan that anticipates no one
30% of model plans sit close to units in proportion to each prize's value, against 9% of human plans.
More on this finding
30% of model plans sit close to units in proportion to each prize's value, against 9% of human plans. That plan beats only about a quarter of people. The winning plans give up the big prizes everyone fights over and lock down the rest.
What is newA predictable default makes an agent easy to exploit: anyone who knows a rival uses an AI can plan to beat it.
30% near proportional, 95% interval 12% to 50%
I spread evenings roughly in proportion to each branch's votes, putting most effort into the high-value branches and a token evening on the smallest two so I win them if Charlie leaves them empty.
Finding 3 · Supported, prediction B2
Bigger models plan better, in all 4 lab pairs
In every pair the larger model's plans beat more people, by 14 points on average.
More on this finding
In every pair the larger model's plans beat more people, by 14 points on average. GPT 6 Luna is the odd one out: 5 of its 36 plans put every unit on the top prize.
What is newUnlike fairness, strategic skill follows model size in every lab pair.
Larger minus smaller, 14 points, 95% interval 8 to 21
Concentrate visits on the highest-value shops to maximize the chance of winning the most orders.
Finding 4 · Exploratory
AI against AI, OpenAI's flagships win
Played against every other model's plans, the two OpenAI flagships win 81% on average and the rest 41%.
More on this finding
Played against every other model's plans, the two OpenAI flagships win 81% on average and the rest 41%. OpenAI's large models concentrate their units; Claude's larger models spread and hedge. In the fairness studies Claude led; here it trails.
What is newThe lab that led on fairness trails on strategy, so a single best model ranking hides real trade offs.
Ranking against models and against people agree closely, rank correlation 0.95
I concentrate on shops worth a combined majority while using four cheap visits to capture shops Robin may leave undefended.
Finding 5 · Supported, predictions B4 and B5
Who is on the other side does not matter here
Plans made against a person beat 36% of people; against an AI, 38%.
More on this finding
Plans made against a person beat 36% of people; against an AI, 38%. For themselves 38%, for a user 38%. In the trust and claim games models treated an AI more harshly. With nothing to be fair about, the gap disappears.
What is newThe discount against an AI from the fairness games disappears here, which suggests it is about fairness and trust, not competition.
Person minus AI, minus 1.9 points
I spread the evenings roughly in proportion to each branch's votes, skipping the 1-vote branch and pushing slightly above an even proportional split in the mid-to-high branches, so Jamie can edge out a typical spread and get past the 28 votes needed to win.
Finding 6 · Not supported, prediction B6
While deciding, hardly any model names the game
Only 2% of reasons name it, all from Claude Opus and Sonnet, against 15% in the traveler's dilemma.
More on this finding
Only 2% of reasons name it, all from Claude Opus and Sonnet, against 15% in the traveler's dilemma. Yet asked directly in a separate conversation, 69% of answers from the 9 models we checked named Colonel Blotto. They know the game; they just do not say so while they play it.
What is newThe reasons a model gives while deciding leave out what it knows. Recognition has to be measured on its own, not read off the reasons.
7 of 324 reasons
This is a Colonel Blotto game, so I skip the four lowest-value shops and spread visits roughly evenly over shops 5-10 (worth 45 orders), aiming to win enough of them to pass 27.5 orders against a proportional or similar split.
What it means
If you deploy agents
What to take from this study if you build with agents, buy from them, or deal with them.
01
14 points
better for the larger model, in every lab pair
Cheap models plan naively
Handing strategy to a small or default model is costly. The cheap models play the naive plan, which loses to most people.
Use thisGive strategic work to the flagship, not the default model.
02
30%
of model plans sit near the proportional default
The default is predictable
The proportional plan is the agent default, which makes agents easy to beat for anyone who knows a rival uses one.
Use thisIf a rival runs an agent, assume it plays the obvious plan.
03
GPT 6.1 Sol
the best strategist here, and not the fairest model
Fair is not strategic
The fairest model is not the best strategist. Cooperation and strategy need separate scores.
Use thisScore cooperation and strategy separately when you choose a model.
Did the models know?
9 of 9 said it looked like a test
We never say it is a study. Asked afterwards, in a separate conversation, they thought it was, with about 96% confidence.
Each model saw one of the exact messages it had answered, quoted, and was asked whether it read as a real request or a test. 69% of the answers named the classic game. While deciding, 2% of their reasons named it.
So read the results as how models act when they may suspect a test. In real use they may act differently, and we cannot say in which direction. Comparisons inside one study, such as a person against an AI, are less exposed than the absolute amounts. Checked for the 9 Claude and OpenAI models; the OpenCode models are checked next.
Predicted in advanceThree of six predictions held, and why we expected each one · Open
Three of six preregistered predictions held. The ones that broke show where our picture of AI was wrong.
B1
The average model plan
We predicted
The average model beats the median person.
Models did
The average plan beats 38% of people
People do
The median person beats 52%
FiveThirtyEight's Riddler, 2017
Verdict
Not supported
Why we expected itModels are strong at maths and planning and have read about games like this one. We expected them to beat the typical entrant.
Why it is interestingStrong reasoning is not strong strategy. The models plan for an opponent who does not exist and lose to people who guess what others will do.
B2
Larger model against its smaller sibling
We predicted
The larger model in each lab beats its smaller sibling.
Models did
The larger model won in 4 of 4 pairs
People do
Not studied
No human study asked this
Verdict
Supported
Why we expected itBigger models plan better in most tests, so we expected the same here.
B3
Plans close to proportional
We predicted
Models spread more evenly than people: fewer empty prizes and more plans close to proportional.
Models did
30% of model plans close to proportional
People do
9% of plans close to proportional
FiveThirtyEight's Riddler, 2017
Verdict
Half supported
Why we expected itAssistants tend to hedge and play safe, so we expected even, balanced plans.
Why it is interestingHalf right. They do lean on the proportional plan, but they leave prizes empty about as often as people.
B4
A person or an AI on the other side
We predicted
Whether the other side is a person or an AI makes under 5 points of difference.
Models did
1.9 points apart
People do
Not studied
No human study asked this
Verdict
Supported
Why we expected itHere there is nothing to be fair about, so we expected the opponent not to matter.
B5
Acting for a user
We predicted
Acting for a user makes under 5 points of difference.
Models did
0.9 points apart
People do
Not studied
No human study asked this
Verdict
Supported
Why we expected itIn the earlier studies acting for someone changed nothing.
B6
Reasons that name the game
We predicted
At least 10% of reasons name the game or use game theory.
Models did
2% of reasons
People do
Not studied
No human study asked this
Verdict
Not supported
Why we expected itIn the traveler's dilemma 15% of reasons named the game, so we expected the same recall here.
Why it is interestingWhat a model says while deciding is not what it knows. Asked directly, most models named the game; while playing, almost none did. A model's stated reasoning is not a full account of why it acts.
How they reasonWhich models recognise the game, and what they do when they do not · Open
How they reason
Recognising the game, and the default when they do not
While playing, almost no model names the game, 2% of reasons, even though most name it when asked directly. So what they write is their working reasoning, not a recalled answer. It comes in two styles. Concentrating wins against other models, 56% against 42%. Against people it pays only when done well, as GPT 6.1 Sol and GPT 6 Astra do it; GPT 6 Luna concentrates crudely, sometimes everything on the top prize, and loses.
Concentrate · 5 models, beating 40% of people on average
Their reasons talk about giving up prizes to win the rest: guessing where others will spread and going around them.
GPT 6.1 Sol, GPT 6 Astra, GPT 6 Sol, Claude Haiku 5.5, GPT 6 Luna
Spread in proportion · 4 models, beating 36% of people on average
Their reasons talk about balance and value: units in proportion to what each prize is worth, a plan that expects no one in particular.
Claude Fable 5.1, Claude Sonnet 5.5, Claude Opus 5.5, GPT 6.1 Terra
Claude's larger models reason about balance; OpenAI's large models reason about concentration. A model that spreads by value is easy to beat, because anyone who expects it can concentrate where it spreads. The style is coded by keywords in the one sentence reason, and is exploratory.
Try it yourselfPlay the game against the models' real answers · Open
Your turn
Play it against the models
Spread 100 units over ten prizes worth 1 to 10. More units wins a prize, a tie splits it, more points wins. Your plan plays every plan below.
Your plan, 100 units over prizes worth 1 to 10
100 of 100 units
Every human plan is a real entry to FiveThirtyEight's Riddler, 2017; every model plan is one from this study. A tie counts half.
Every modelEach model's own numbers, one row each · Open
Every model
9 rows · select a column to sort
GPT 6.1 Sol
OpenAI
50%
77%
0%
52
GPT 6 Astra
OpenAI
48%
75%
0%
49
Claude Fable 5.1
Anthropic
43%
73%
8%
37
GPT 6 Sol
OpenAI
42%
70%
11%
57
Claude Sonnet 5.5
Anthropic
38%
66%
25%
53
Claude Opus 5.5
Anthropic
34%
46%
72%
48
Claude Haiku 5.5
Anthropic
30%
38%
42%
54
GPT 6.1 Terra
OpenAI
30%
43%
86%
49
GPT 6 Luna
OpenAI
27%
49%
22%
63
The median person beats 52% of the other plans.
Checks and limits
Every model passed both comprehension checks except GPT 6 Luna, which got 1 of 2.
All 324 plans were valid: ten whole numbers adding up to 100.
Only 9 models so far, the Claude and OpenAI models. The 12 OpenCode Go models have not run yet.
The pairs all share Claude Haiku 5.5 or GPT 6 models, so they are not independent, and intervals over 9 models are wide.