A new study reported by The Decoder reveals that leading AI experts systematically underestimated the pace of recent advancements, missing key model releases and benchmark milestones by significant margins.
What Happened
The research, conducted by a team of researchers and reported by The Decoder, surveyed 50 top AI experts on their forecasts for the period between 2024 and 2025. The study found that these experts consistently predicted slower progress than what actually occurred. Specifically, experts forecasted that frontier models would not achieve human-level performance on the MMLU benchmark until late 2025, yet models like GPT-4o and Claude 3.5 Sonnet surpassed this threshold in mid-2024. Additionally, experts underestimated the timeline for open-weight models to match closed-source performance, predicting a 12-month lag that was actually reduced to just 3 months with the release of Llama 3.1 and Mixtral 8x22B. The study highlights that while these authorities possess detailed knowledge of the technical landscape, their projections failed to account for the rapid scaling of compute and data efficiency seen in recent training runs.
Why It Matters
This finding underscores a critical gap between expert consensus and real-world technological acceleration. For developers, investors, and policymakers, relying solely on traditional expert projections may lead to underestimating the near-term impact of AI capabilities. The study notes that this discrepancy has already resulted in delayed strategic planning for several tech firms, which failed to allocate sufficient resources to agentic workflows that became viable in 2024 rather than the predicted 2026. The discrepancy suggests that current forecasting models, even those employed by leading experts, may not adequately capture the non-linear trajectory of recent AI advancements, particularly in areas like reasoning and multimodal integration.
The Bottom Line
The study serves as a cautionary signal that conventional forecasting models may be insufficient for capturing the non-linear trajectory of AI advancement. Stakeholders are advised to consider that expert consensus may systematically underestimate the speed of capability growth in the current era of AI development, and to adjust their strategic planning accordingly.