LLM judges are scorers, not oracles

Using a model to grade a model is useful and dangerous in the same breath. The failure mode is treating the judge as an oracle instead of an instrument.

Instruments drift

A judge behind a moving alias silently changes your scores. In Omic, judges are pinned and versioned like any other scorer — change the judge, and your baseline is invalidated on purpose.

Deterministic first

Most checks don’t need a judge. Exact match, contains, and similarity cover more than teams expect, and they never disagree with themselves. Reach for a judge only when a rule genuinely cannot express the criterion.

When you do, write the rubric down. A judge with no rubric is a vibe with an API bill.