Stanford researchers Owen Queen and James Zou have found that 44 percent of widely cited research claims about large language models have already expired. The researchers tested 50 claims against 20 models released between July 2024 and September 2026, revealing that the average lifespan of findings with an estimated expiration date was just 15.7 months. The study showed that only 16 percent of findings still hold true, while 34 percent have weakened significantly and 6 percent have strengthened.
The research demonstrated that claims regarding what models cannot do expire the fastest, particularly reasoning-related weaknesses like counting letters or performing arithmetic in unfamiliar number systems. In contrast, findings about behavioral preferences and tendencies proved to be longer-lived, including sycophancy, self-evaluation preferences, and political biases. The study also highlighted wide performance gaps between models from the same release window, noting that neither release date, provider, nor model size reliably predicts the observed patterns.
Queen and Zou concluded that empirical findings about artificial intelligence should be framed as point-in-time observations of specific models rather than universal truths. They recommended that academic papers thoroughly document model versions, exact dates, prompts, and analysis code to enable continuous re-testing on newer models. This rapid evolution highlights the challenge of maintaining accurate scientific literature in a fast-moving technological landscape.