Know if your AI got better
Omic covers the path from a single eval run on your laptop to a gated CI pipeline your whole team trusts. One data model, one CLI, one file format — whether you’re running locally or on Cloud.
It’s comprised of Omic SDK: an open-source CLI and library for running, comparing, and storing evals; and Omic Cloud: a hosted run history with CI gating and shared access for teams.
The problem
Shipping AI means changing prompts, models, and retrieval constantly. But teams can’t move fast because:
•
There’s no reliable way to know if a change made things better or worse
•
Eval results live in notebooks, spreadsheets, or someone’s terminal history
•
Comparing two versions means re-running everything manually and eyeballing the diff
•
CI has no way to gate a merge on behavior, only on tests that pass or fail
The best AI teams ship faster because they trust their evals. Omic gives you that trust in one command.
Who is Omic for?
Omic is built for teams shipping AI they can’t afford to guess about:
•
AI engineers iterating on prompts, chains, and agents
•
ML engineers tracking model quality across versions
•
Platform teams adding eval gates to CI pipelines
•
Anyone shipping AI who’s tired of guessing whether the new version is better
If you’re changing prompts, swapping models, or tuning retrieval, Omic tells you what moved and by how much.
How do you use it?
Omic follows one loop from a local run to a gated merge. Every stage speaks the same data model, so nothing needs re-plumbing between them.
Run
Define a suite of cases and scorers. Run them locally with omic run. Get a verdict in seconds.
Compare
Every run is automatically diffed against your baseline. See which cases improved, which broke, and the exact outputs that changed.
Understand
Query across your full run history. Find when a metric started sliding. Trace a regression back to the commit that caused it.
Ship
Add one line to your CI config. If a suite drops, the merge stops. The failing cases are in the PR comment.
Get started
Install the SDK and run your first eval in two minutes, or browse the docs to understand the concepts first.
Can’t find what you’re looking for?
Search the docs with /, ask on Discord, or open an issue on GitHub.