Know if your AI got better
Open-source evals for your AI apps. Run them on your laptop while keeping every result, & compare any two versions to get proper analytics.
@omic-ai/omic
Compare any two versions
Diff any run against any baseline. See which cases improved, which broke, and the exact output that changed.
Run anywhere
The same command on your laptop, in CI, or against production traffic. No account, no queue, no waiting on someone else's cluster.
Keep every result
Every run is an .eno file you own. Commit it, grep it, or hand it to Cloud when one machine stops being enough.
Public by default
We run Omic on Omic. Every result, including the ones that got worse.
sql-correctness
0.71 ▼ 0.09
3f2a1c ✗
groundedness
0.94 ▲ 0.06
3f2a1c ✓
retrieval@5
0.83 ▲ 0.11
9d41f0 ✓
citation-accuracy
0.66 ▲ 0.21
2j4ee9 ✓
latency-p95
0.86 ▲ 0.13
4f37a3 ✓
hallucination-rate
0.62 ▼ 0.13
3f2a1c ✗
inline-comment
0.07 ▲ 0.55
4f37a3 ✓
tone
0.88 -
9d41f0 ✓
One command.
Run every eval you have, compare against any baseline, and see exactly what changed.
A tool you run, not a dashboard you log into. Your evals, your machine, your files.
One history for every run your team makes
The hosted backend for Omic. Runs where your code runs.
Keep
Every run, not just the last two
Your laptop keeps the folder. Cloud keeps the record.
Compare
Compare across the whole history
Diff runs locally with the CLI. Cloud shows your full history.
Gate
Catch the regression before it merges
One PR check. If a suite drops, the merge stops and the failures are in the diff.
Share
Everyone argues from the same run
Send a link, not a screenshot — for reviews, postmortems, and future questions.
News
The eval that doesn’t lie to you
With the 0.4 release, Omic ships baseline diffing, CI gating, and a run history you own end to end. This post covers why we made .eno a file instead of a database row, walks through the new comparison engine, and introduces Omic Cloud — our hosted run history for teams.
The community loves building with Omic
Omic is open source and runs in the CI of teams shipping AI to real users.
Jared
@jjarde
Omic is the first eval tool i’ve actually kept installed. no account, no dashboard, just `omic run` and it tells me if i broke something.
Sans Ms
@sansmss
unreasonable amount of joy from watching a number go up.
Rull Azen
@rullazen
Migrated our suite over on Friday and hit one rough edge with the custom scorer API, but a maintainer replied on Discord in about ten minutes so I’ll allow it.
Kev
@kevshipss
The fact that their own homepage shows a failing eval is the only reason i trusted the other two numbers.
tomOor
@orr09
You dont need a platform to know if your prompt got worse, you need a cli and a baseline.
jsaill
@j_sail
ok this rules.
Wuww
@wuww77
i am not a “put it in CI” person i am a “run it at 2am and stare” person and it works fine for that too.
Venesis
@venesize
watched it fail my own suite on the first run and honestly? refreshing
Zhaoxing L
@zhaoxll
My eval results are in git now, where they belong.
