Python SDK
Everything the CLI does is available in Python. Use the SDK when you want to build suites programmatically, or run evals inside a pipeline you already have.
Defining a suite
Build cases in code:
•
omic.Suite() — a group of cases that run together
•
suite.case() — add an input and its expectation
•
@omic.scorer — register a custom scorer
Running and comparing
Run a suite and read the result:
•
suite.run() — returns a Run object
•
run.compare(baseline) — the per-case diff
•
run.save(path) — write a .eno file
Working with results
Results are plain objects:
•
run.score — the aggregate score for the suite
•
run.cases — every case with its output and score
•
run.regressions — just the cases that got worse