Findings

This page explains each line of a finding, how rb reads the runs, and the integrity problems a finding can carry.

rb never stores a finding, and no finding is one word. Each time the record is read, rb works out every criterion’s finding from its runs and analyses, and says it on separate axes. This is rb show of a criterion after five seeds:

$ rb show interval-within-budget
Criterion interval-within-budget on drop (primary, own): The whole 95% interval of dropout's cost stays within 0.02 nats · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout"
  interval-within-budget The whole 95% interval of dropout's cost stays within 0.02 nats
          criterion met · unresolved · change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5)
          reads:     change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)
          result:    unresolved: the interval includes 0
          rule:      criterion met: the interval's worse bound +0.01338 inside 0.02
          timing:    declared before the evidence (by production times)
          units:     5 seeds
          integrity: ok
  runy2bsxn: difference +0.008 (file) · unit seed=1 · seed=1
  run5hv5cu: difference -0.003 (file) · unit seed=2 · seed=2
  runaxeil6: difference +0.014 (file) · unit seed=3 · seed=3
  runkeudey: difference +0.003 (file) · unit seed=4 · seed=4
  runkxmrdp: difference +0.006 (file) · unit seed=5 · seed=5

interval-within-budget is the criterion’s id: its name in every command.

The axes

Line	What it says
reads	The comparison rb read: the number, its interval, its unit, how it was computed, and from what. (rb) is rb’s reading of the runs; an analysis is named by its id.
result	What the number shows, in the metric’s direction. gain or loss: the interval is clear of 0 on the better or the worse side. unresolved: the interval includes 0. equivalent: inside an --equivalent margin. point only: no interval yet. measured: one arm on its own, with its interval.
rule	Whether the criterion’s rule is met on that reading: criterion met, criterion not met, criterion undetermined (it needs an interval, more units, or the metric’s direction), or criterion untested. With no interval, or with an interval that crosses the threshold, a rule that holds reads estimate meets criterion. A cited criterion reads reproduced or not reproduced, and reproduced on the estimate where it is met on a reading with no interval or one that crosses its line; a gate, gate met or gate missed. The tally says how many rules are met on the estimate alone: “Rules: 5 reproduced, 2 of them on the estimate alone”.
timing	Whether the criterion was declared before its evidence. See Criteria.
units	How many independent units the reading rests on, and what a unit is: here, a seed.
integrity	Every problem with what the finding depends on, or ok.

The result and the rule are separate on purpose. “criterion met · unresolved” above is an ordinary finding: the rule asked that the whole interval of dropout’s cost stay within 0.02 nats, and it does, while the interval still includes 0, so rb cannot say dropout changes the loss at all.

How rb reads the runs

• Each run gives one difference: the subject’s number minus the reference’s, both from that run.
• Runs with the same per-run values (two runs at seed=2) are one unit, their numbers averaged. A run recorded with --repeats <run> checks another run, and adds no unit.
• The estimate is the mean over units. From 3 units, rb gives a 95% Student t interval and labels it as its own.
• Arms recorded in separate runs pair up by unit. The reading says what it pairs them by (“paired by seed”), with the unpaired reading beside it, unless it says exactly the same. Whether runs that share a seed or an epoch are linked is yours to say: rb never decides it. Arms that never share a unit are compared unpaired (Welch).
• A setting that only numbers repeats, such as an epoch, is declared with rb spec set <exp> population.repeat epoch --no-verify (or seed,epoch). No reading is then said to pool it, a rule is never read at one of its values alone, and it is never offered as the unit.
• A run is shown and not counted, with the reason, when its config disagrees with the spec, when it was recorded under an earlier spec, or when it was edited outside rb.

Why one number is not enough

With one or two seeds there is no interval, and a rule on the estimate can still hold: rb says estimate meets criterion · point only. That is not “it works”. These are two criteria on the same comparison, one on the estimate and one on the interval, after the first seed:

$ rb run add drop --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"
runy2bsxn on drop: none.val_loss=1.31, p01.val_loss=1.318 · seed=1
  Ran with the spec: 3 of its settings found in out/seed1/config.yaml, all matching.
  Produced: 2026-11-06 20:25:24 (file)
  mean-within-budget estimate meets criterion · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)
  interval-within-budget criterion undetermined · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)
  rb put the code in git at refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96 removes it.

Next: rb show mean-within-budget  ·  rb show interval-within-budget

And after the third: the interval appears, but it still reaches past 0.02, and the second criterion asked for 5 seeds:

$ rb run add drop --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"
runaxeil6 on drop: none.val_loss=1.315, p01.val_loss=1.329 · seed=3
  Ran with the spec: 3 of its settings found in out/seed3/config.yaml, all matching.
  Produced: 2026-11-06 20:25:25 (file)
  mean-within-budget estimate meets criterion · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)
  interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)

Next: rb show mean-within-budget  ·  rb show interval-within-budget

Get an interval, not a single number walks through all five seeds.

Integrity

A finding lists every problem it depends on. The common ones:

Problem	What it means
confound	the two arms differ in a setting the experiment does not say it varies, and nobody accepted the difference
conflict, stale	a setting’s source states another value, or no longer states it
unknown	a required setting nobody has found
not_comparable	a setting differs from the value a cited source used, or a cited difference of means is one rb’s paired reading cannot give
drift	a frozen experiment no longer matches its freeze
edited_outside_rb	a record it depends on was changed by hand

A setting that is provisional, not verified, is a note on the finding, not a problem. So are numbers typed in, a run with uncommitted changes, arms that differ in a per-run setting (500 problems against 250), and runs that pool several values of a setting. A rule’s card on the page rb page writes carries the same notes as rb show.

In CI

rb status --fail-on <gate> exits 1 when the record fails that gate. unestablished is the one most CI jobs want: it fails when a primary criterion is not met, undetermined or untested, when it is not shown to have been declared before what it reads, when a finding has an integrity problem, or when a record was changed outside rb. Check out the whole history in CI (fetch-depth: 0 with actions/checkout): rb tells a clean merge from a hand edit only through git’s history.

rb status --fail-on unestablished

The other gates are unmet, unresolved, untested, integrity, late, undecided, stale, checks and unknown. rb docs says what each one checks.