Evals against production traffic, safely

The best eval cases are the ones your users already sent. The trick is using them without turning your eval pipeline into a privacy incident.

Sample, scrub, freeze

Sample a slice of production requests, scrub identifiers at the edge, and freeze the result into a suite. The suite is a file in your repo — reviewable, versioned, and stable while you iterate.

Shadow runs

Run the frozen suite against a candidate model before it ships. Nothing touches production; you are replaying the past against the future.

When the diff is clean, promote. When it isn’t, the failing cases are real user traffic — the most honest regression report you can get.