Foundations
What is Omic?
Here is a situation every AI team knows. Someone changed a prompt last week and the app feels worse. The evidence is a few screenshots, a spreadsheet nobody kept updating, and three people who disagree. By the time you can prove it, the change is four commits deep.
Omic exists so that argument stops happening. It is one way to run evals, keep every result, and compare any two versions — on your laptop, in CI, or against production traffic.
Run
One command, every suite.
An eval starts as a suite of cases — an input, and what you expect back. Omic runs them all with a single command, scores each one, and writes the result to a file you own.
Run from the CLI or the Python SDK. The same command works on your laptop, in CI, or against production traffic. No account, no queue, nothing to stand up first.
Compare
Diff against any baseline.
Once a run exists, compare it against any other run. Omic diffs case by case, so you see which cases improved, which broke, and the exact output that changed — never a single blended score.
Pick a baseline by branch or by commit. The comparison works the same whether you are checking a prompt tweak locally or gating a pull request in CI.
Understand
The history is the point.
A regression rarely announces itself. Query across your full run history to find when a metric started sliding, then trace it back to the commit that caused it.
Every run is an .eno file holding the cases, the outputs, the scores, and the commit that produced them. Nothing is locked away — read it with the CLI, or export it and use your own tooling.
Ship
Gate the merge.
Add one line to your CI config. If a suite drops against its baseline, the merge stops and the failing cases show up in the pull request.
Because running, comparing, and gating all speak the same data model, there is no seam between them. The suite you ran on your laptop is the suite that guards main.
Make it yours
That’s the whole loop: run, compare, understand, ship, and back again. Install the SDK, point it at a suite you already have, and see your own numbers in a couple of minutes.
Start with the Quick start, read the docs, or open an issue on GitHub. Questions are welcome in the community on Discord.