Business IEA Says Data Center Power Demand Could Double to 1,000 TWh by 2030, With AI Supplying Three Quarters of the Growth Chatbots ChatGPT Now Transcribes as You Talk: OpenAI Removes the 10-Minute Cap and Streams Words Live on Every Plan Business Ex-Alibaba Executive's Everful Raises $48.5 Million to Put AI Agents in Charge of Sourcing From 30,000 Factories AI Agents Google Built a Mirror Dimension of the Internet to Train Its Chrome AI Agents Without Breaking the Real Web Productivity Canva Finally Adds an Off Switch for Its Generative AI Features, and a Filter to Hide AI-Made Templates Business Anthropic Begins Trading Today: 980 Million Shares at $83, a $1.5 Trillion Valuation and the Largest IPO on Record News Google Books Blue Origin New Glenn to Launch TPU-Carrying AI Satellites in Early 2029 Security Hackers Used the AI Penetration Tool ARTEX to Disrupt European Commission Systems for Over Ten Days Business IEA Says Data Center Power Demand Could Double to 1,000 TWh by 2030, With AI Supplying Three Quarters of the Growth Chatbots ChatGPT Now Transcribes as You Talk: OpenAI Removes the 10-Minute Cap and Streams Words Live on Every Plan Business Ex-Alibaba Executive's Everful Raises $48.5 Million to Put AI Agents in Charge of Sourcing From 30,000 Factories AI Agents Google Built a Mirror Dimension of the Internet to Train Its Chrome AI Agents Without Breaking the Real Web Productivity Canva Finally Adds an Off Switch for Its Generative AI Features, and a Filter to Hide AI-Made Templates Business Anthropic Begins Trading Today: 980 Million Shares at $83, a $1.5 Trillion Valuation and the Largest IPO on Record News Google Books Blue Origin New Glenn to Launch TPU-Carrying AI Satellites in Early 2029 Security Hackers Used the AI Penetration Tool ARTEX to Disrupt European Commission Systems for Over Ten Days

Evaluating AI Output: Metrics That Mean Something

Evaluating AI Output: Metrics That Mean Something

Automated scores are useful for tracking regressions, not for deciding whether something is good. Human review on a fixed set remains the reference.

Evaluation is where most AI projects quietly fail. Without a fixed test set and an agreed pass criterion, every improvement claim is an opinion.

Building the reference set

Collect real inputs from production, not invented ones. Fifty to two hundred examples is enough to start if they are representative. Label the correct or acceptable output for each. Store it in version control so it can grow without losing history.

Layered scoring

  • Deterministic checks. Does the output parse as valid JSON? Does it contain every required field? Does it stay within a length limit?
  • Rubric scoring. A human rates accuracy, completeness and tone on a small integer scale.
  • Comparative preference. Two outputs, which is better. Easier to judge consistently than absolute scores.
  • Model-as-judge. Cheap and useful for regression tracking, but it must be validated against human labels before being trusted.

Watch for contamination

If the same examples were used to tune prompts, scores drift upward without real improvement. Hold back a portion of the set and never use it during development.

Comments (0)

Log in to join the discussion

Log In

No comments yet