AI for Impact 
Why Measuring AI Is Hard
AI systems routinely achieve impressive scores on standardized tests and benchmarks, yet those results often tell us little about whether a tool will succeed in a real government setting.
This session explores the limitations of common AI benchmarks, the differences between predictive and generative AI systems, and why performance in laboratory tests often fails to predict success in real-world settings.
By the end of this workshop, participants will be able to:
- Explain why standard AI benchmarks and test scores may not predict how well an AI system will perform on real government tasks and workflows.
- Distinguish key evaluation challenges for predictive and generative AI systems and identify why each requires different approaches to measuring performance.
- Identify the factors agencies should consider when defining meaningful measures of AI effectiveness, reliability, and public value in real-world government settings.
This workshop is part of an InnovateUS Series called : Practical Approaches to Evaluating AI for Public Benefit
Click here to view all workshops from this series
Scaling Government Innovation
December 10, 2026
Building High-Performing Product Teams
December 4, 2026
The Rise of the Citizen Developer
November 17, 2026
From Projects to Products
November 9, 2026
AI Changes: Buy vs. Build
November 4, 2026
Measuring What Matters
September 15, 2026