Why we publish our own regressions

Every eval vendor shows you a chart where the line goes up. We decided to publish ours whether it goes up or not, including the suites that are currently failing.

The argument for it

A tool that measures quality should be willing to be measured. If we only showed green rows, the honest reader would assume we picked the green ones — which means the green rows would carry no information either.

A visible failing row is the cheapest credibility available to us, and it is the one thing a competitor cannot copy without also publishing their own regressions.

What it changed internally

More than we expected. A number that is visible to strangers gets fixed faster than a number in a private dashboard. Our sql-correctness suite had been quietly broken for weeks before we started publishing it.