Artificial Analysis overhauled its Intelligence Index, widening the performance gap between open and closed models to 12 points up from a previous 3 points. The update caused the Kimi K3.1 Pro model to drop five spots in the rankings, falling from fourth place down to ninth place behind eight closed models. Grok 4.7 also lost 12 points in the updated evaluation.
The changes in version 5 of the index stem from Artificial Analysis relying on private test sets for 60 percent of the evaluation, a shift designed to prevent benchmaxxing where models are specifically trained on known test questions. AI researcher Nathan Lambert noted that training gains also occur in reinforcement learning environments with tasks resembling benchmarks, and keeping up across multiple domains requires sophisticated infrastructure that not every lab possesses.
While the new private test sets help curb optimization for specific questions, they introduce drawbacks because outsiders can no longer spot prompt errors. This limitation was highlighted when Artificial Analysis had to revise ambiguous and flawed questions within its AA-Omniscience knowledge test for the new index version.
This index update matters because it provides a more rigorous evaluation of artificial intelligence capabilities by curbing benchmark optimization and revealing true performance disparities between open and closed models.