The eval that doesn’t lie to you

With the 0.4 release, Omic ships baseline diffing, CI gating, and a run history you own end to end. This post covers why we made .eno a file instead of a database row, walks through the new comparison engine, and introduces Omic Cloud — our hosted run history for teams.

Why a file, not a database row

Most eval tools write results into a service you rent. That is convenient right up until you want to leave, diff two runs offline, or check a result into the repo next to the code that produced it. We wanted the opposite default: a run is a file, the file is yours, and the tool is optional.

An .eno file holds the cases, the outputs, the scores, and the commit that produced them. You can commit it, grep it, or hand it to Cloud when one machine stops being enough. Nothing about the format requires us to exist.

The comparison engine

Comparison is per case, never a single blended number. A run that averages the same as its baseline while quietly breaking fourteen cases is not the same run, and a single score hides that. Omic reports what improved, what regressed, and the exact output that changed.

Picking a baseline

A baseline is just another run. Pick it by branch to compare against the latest run on main, or by commit to pin the comparison to an exact revision. The same command works locally and in CI.

What’s next

Cloud is in early access. The SDK stays Apache-2.0 and MIT, and it stays free — the paid product is the shared history, not the tool.