Suites & cases
A suite is a group of cases that belong together. A case is one input and the expectation attached to it. This is how evals stay organized once you pass a handful of examples.
Suites
A suite is the unit you run and gate on:
•
One concern per suite — retrieval quality, tone, refusals — keep them separate.
•
Runs as a unit — a suite passes or fails as a whole.
•
Versioned with your code — suites live in the repo, next to what they test.
Cases
A case is the smallest thing Omic can score:
•
An input — the prompt, question, or payload you send.
•
An expectation — an answer, a rule, or a reference to compare against.
•
Metadata — tags you can filter and group by later.
Organizing at scale
Once you pass a few dozen cases:
•
Tag by behavior — so a regression points at what broke, not just where.
•
Split slow cases out — keep the fast suite gating every pull request.
•
Keep cases small — one case should test exactly one thing.