I was wrong and the AI corrected me

At the start of the year I introduced my AI benchmark for 2026, or at least the first one I was planning to run. That post contained a mistake and this is why the 80 models tested not so well on the first problem. I was recently performing the analyses for a followup on that competition when one of the models actually insisted that I was wrong and showed me the mistake. It turns out that I was indeed wrong – can you spot the mistake in that first article?

The mistake 🔗

In the results article I said I triple checked the math for the square grid problem – arrange the numbers in a square spiral and sum the neighbors of 2026. And I did, I checked that the corners follow the correct pattern and that the sum of the neighbors is what I was looking for. That is, I first checked that the bottom right corner contains squares of odd numbers:

17 16 15 14 13
18  5  4  3 12
19  6  1  2 11
20  7  8  9 10
21 22 23 24 25 ...

Indeed, 1, 9, 25 are squares of odd numbers, and so are 2025 and 1849. This is why I said that the neighborhood of 2026 is the following:

... 1850 2027 2212 ...
... 1849 2026 2211 ...
... 2024 2025 2210 ...

Indeed, as I triple checked, the sum of these numbers is 16408. And this is the number I was expecting as the solution. But, do you see the mistake?

After the square of the odd number I went up instead of to the right. In the first spiral above, 2, 10, and 26 are all to the right of the square corner, but in this neighborhood, 2026 is upwards. The correct grid is:

... 1850 2027 2212 ...
... 2025 2026 2211 ...
... 2208 2209 2210 ...

And now, the correct answer is 16952. Now, we need to regrade all the LLMs, they were not as bad as I said.

The correct results 🔗

I will jut copy the tables from the results article, but corrected. I’ll mark with bold the numbers that have been updated (only in the square column):

The score of all models on the square problem has increased, in some cases quite significantly. There are only a few cases where the hexagon problem is resolved better than the square one, and this seems to match the cases where the model cheats and finds the corresponding OEIS sequence – will talk about this in a future article.

Plotting the model performance across the two problems, we also see a different spread:

The costs, however, stay the same as in the previous table. But, since the performance on the square problem changed, the cost/performance graph for that problem also changes:

This means that the plot of cost/performance across both problems changes:

And this implied we get a new Pareto frontier:

As a list, the models that are on the frontier are:

• grok-4-fast (score: 20.0, cost: $0.132)
• openai/gpt-oss-120b (score: 16.0, cost: $0.02)
• grok-4-fast-non-reasoning-latest (score: 1.0, cost: $0.006)

This time, the last model on the Pareto frontier is another Grok model. It got one correct answer and is cheaper than gemini/gemini-2.0-flash-lite by 0.2 cents, so it dethrones it.

Overall, it seems that besides giving some expensive models their just score, the Pareto frontier barely moves.

Conclusions for future experiments 🔗

I really need to be more careful in the future with these puzzles. If I’m asking the model to be correct 100%, maybe I should first make sure that I expect the correct answer. I should always check all the aspects of the problem, not just the arithmetic ones.

Hope the LLMs forgive me for asking them to be correct when I was the one expecting them to provide the answer that I got to after a human mistake.

PS: I updated my demos repo with the new results and analysis. Next, I’ll also add a correction note to the other two articles.