Business IEA Says Data Center Power Demand Could Double to 1,000 TWh by 2030, With AI Supplying Three Quarters of the Growth Chatbots ChatGPT Now Transcribes as You Talk: OpenAI Removes the 10-Minute Cap and Streams Words Live on Every Plan Business Ex-Alibaba Executive's Everful Raises $48.5 Million to Put AI Agents in Charge of Sourcing From 30,000 Factories AI Agents Google Built a Mirror Dimension of the Internet to Train Its Chrome AI Agents Without Breaking the Real Web Productivity Canva Finally Adds an Off Switch for Its Generative AI Features, and a Filter to Hide AI-Made Templates Business Anthropic Begins Trading Today: 980 Million Shares at $83, a $1.5 Trillion Valuation and the Largest IPO on Record News Google Books Blue Origin New Glenn to Launch TPU-Carrying AI Satellites in Early 2029 Security Hackers Used the AI Penetration Tool ARTEX to Disrupt European Commission Systems for Over Ten Days Business IEA Says Data Center Power Demand Could Double to 1,000 TWh by 2030, With AI Supplying Three Quarters of the Growth Chatbots ChatGPT Now Transcribes as You Talk: OpenAI Removes the 10-Minute Cap and Streams Words Live on Every Plan Business Ex-Alibaba Executive's Everful Raises $48.5 Million to Put AI Agents in Charge of Sourcing From 30,000 Factories AI Agents Google Built a Mirror Dimension of the Internet to Train Its Chrome AI Agents Without Breaking the Real Web Productivity Canva Finally Adds an Off Switch for Its Generative AI Features, and a Filter to Hide AI-Made Templates Business Anthropic Begins Trading Today: 980 Million Shares at $83, a $1.5 Trillion Valuation and the Largest IPO on Record News Google Books Blue Origin New Glenn to Launch TPU-Carrying AI Satellites in Early 2029 Security Hackers Used the AI Penetration Tool ARTEX to Disrupt European Commission Systems for Over Ten Days

Building a Small Evaluation Harness Before You Ship

A fixed test set, a scoring script and a changelog. Half a day of work that makes every later decision cheaper.

Most AI features ship without a way to tell whether a change helped. Building that first is the highest-return half day in the project.

Components

  • Cases file. Inputs and expected outputs, stored as JSON in the repository.
  • Runner. Executes the current system against every case and writes results to a file.
  • Scorers. Deterministic checks first — schema validity, required fields, forbidden phrases — then a rubric score for the rest.
  • Report. One line per case plus a summary, designed to be readable in a terminal.

Run it on every change

Prompt edits, model version bumps, retrieval tweaks. Recording scores over time turns "it feels better" into a trend line, and it catches regressions that a spot check misses.

Keep a holdout

Reserve part of the case set and never use it during development. Otherwise you tune against your own test set and the scores stop predicting real performance.

Start small

Twenty well-chosen cases beat two hundred scraped ones. Add a case every time production surprises you — that is how the set stays relevant.

Comments (0)

Log in to join the discussion

Log In

No comments yet