Scorers
A scorer turns an output into a number. Everything Omic reports — passed, failed, improved, regressed — comes from scorers, so choosing them well matters more than any other decision you make.
Built-in scorers
Enough to cover most suites:
•
exact — string equality, for deterministic outputs
•
contains — checks that a required substring is present
•
similarity — embedding distance against a reference answer
LLM judges
For when correctness is a judgement call:
•
Rubric-based — grade against criteria you write in plain language.
•
Pinned model — the judge is versioned, so scores stay comparable.
•
Cached — identical inputs are not re-judged, or re-billed.
Choosing a scorer
The rule of thumb:
•
Deterministic first — reach for a judge only when a rule genuinely cannot express it.