Open-source evals for your AI apps
Use the Omic SDK to define suites, run them anywhere, and compare any two versions against a baseline. One CLI, one file format, and results you keep.
A single toolchain
for every stage of the loop
Keep
Every run, not just the last two
Your laptop keeps a folder. Cloud keeps the record — every run, every commit, every teammate, going back as far as you've been shipping.
Compare
Compare across the whole history
Diff any two runs locally with the CLI. Cloud lets you diff across every commit at once, so you can see when a metric started sliding instead of noticing six weeks late.
Gate
Catch the regression before it merges
One check on the PR. If a suite drops against its baseline, the merge stops and the failing cases are right there in the diff.
The community loves building with Omic
Omic is open source and runs in the CI of teams shipping AI to real users.
Jared
@jjarde
Omic is the first eval tool i’ve actually kept installed. no account, no dashboard, just `omic run` and it tells me if i broke something.
Sans Ms
@sansmss
unreasonable amount of joy from watching a number go up.
Rull Azen
@rullazen
Migrated our suite over on Friday and hit one rough edge with the custom scorer API, but a maintainer replied on Discord in about ten minutes so I’ll allow it.
Kev
@kevshipss
The fact that their own homepage shows a failing eval is the only reason i trusted the other two numbers.
tomOor
@orr09
You dont need a platform to know if your prompt got worse, you need a cli and a baseline.
jsaill
@j_sail
ok this rules.
Wuww
@wuww77
i am not a “put it in CI” person i am a “run it at 2am and stare” person and it works fine for that too.
Venesis
@venesize
watched it fail my own suite on the first run and honestly? refreshing
Zhaoxing L
@zhaoxll
My eval results are in git now, where they belong.
