Eval Scorecard

Demo & usage

More examples

When to use

• Judging whether a prompt or model change actually regressed anything, rather than reading a headline pass rate.
• An eval board people are expected to act on — where a false regression costs more than a missed one.
• For per-run token cost and latency, use the AI Token Dashboard; this is about correctness.

Async states

• loading
• error
• retry

Outputs

Wire it to your API

This component is headless about transport: it renders what you give it and reports what the user asks for. Nothing below requires @styloviz/data — drive it from whatever you already use.

• This component does not emit a query, so it renders what you give it. Fetch on your own trigger and bind `history`, `loading` and `error`.
• Normalize failures with `toSvDataError()` from `@styloviz/core`. It derives `retryable` from the status, so a 404 does not get a "Try again" button that can only fail again.

Full walkthrough: Connecting to a backend.

Customization & theming

• `run`, `baseline` and `history` are inputs and all default to empty/null; import `DEMO_RUN` / `DEMO_BASELINE` for a demo.
• Three layouts via `variant`: `summary` (per-suite rates, worst first), `compare` (the verdict plus a case-by-case list) and `cases` (this run’s cases, failures first).
• Pass rates carry a 95 % Wilson interval — the Wilson form because eval suites are small and rates sit near 0 or 1, where the textbook normal interval returns bounds outside 0–100.
• `error` and `skipped` cases are excluded from the denominator and surfaced as an "ungraded" count: an error is the harness falling over, and counting it as a failure blames the model for the infrastructure.
• The delta is only coloured when the two rates’ intervals do not overlap. Everything else stays grey, because a red arrow on noise is what teaches a team to stop reading its own eval board.
• The case list sorts broken → fixed → added → removed → stable, and `caseSelect` emits the delta the reader activated.
• A scale ceiling: `[maxCases]` (300) bounds both case tables. The window is taken AFTER the worst-first sort, which is what makes it tolerable — the failures are at the top and the rows dropped are the passes; slicing first would keep an arbitrary subset and then arrange it beautifully with the failures left outside. The pass rate is still computed over the whole suite. `0` turns it off.
• `provenance` on a run — judge model, judge version, rubric version, seed, trial count, dataset hash and dataset size — is rendered as only the fields the run declared. An eval you cannot reproduce is an anecdote, and a missing seed does not mean 'unseeded': the gaps are named in a **Not reproducible** badge rather than defaulted into something that looks fine.
• The compare view performs the check no eval harness performs: if the two runs used a different judge, judge version, rubric, trial count or dataset, the delta is **a measurement of the judge**, and the component says so above the verdict. A different *seed* is deliberately not flagged — two runs at different seeds are the same experiment sampled twice, which is what you want to compare — and a field only one run declared is a gap, not a mismatch.
• Export: the **Cases (CSV)** button downloads the case list and emits it through `(exportRequested)`, so a host can upload it instead. The provenance rides along in the comment header, because a case list with no judge and no rubric is the anecdote this component exists to stop producing.

Best practices

• Do: Pass a `baseline`. Without one the component reports rates and refuses to report movement, which is right — a run compared with itself prints a confident "no change" over a question nobody asked.
• Do: Pass `history` (three runs or more) so cases that flip in both directions are marked as flaky rather than reported as a regression one day and a fix the next.
• Don't: Do not read the delta without the interval. Three points on fifty cases is one and a half cases, well inside any grader’s variance.
• Tip: Cases present in only one run are reported apart from the delta. A suite that grew by twenty cases has a rate that moved for a reason with nothing to do with the model.

Accessibility

• Each case row carries an accessible name stating what changed, its before/after outcomes, and the flakiness caveat in words.
• Suite bars state their counts and rate as text, not only as a bar width.
• The changed-only filter reports `aria-pressed`.