Get an interval, not a single number

This cookbook runs seeds until a criterion has the interval it asks for, and shows rb’s finding after each seed.

One seed gives one number, and one number cannot tell a real cost from run-to-run noise. Ask for an interval in the criterion itself, declare how many seeds it needs, and rb will not call it before then.

Paste this into your agent

Paste this into your agentCopy

Read the result on its 95% interval, over 5 seeds declared up front.

Use Rabbit Brain to find out whether dropout 0.1 costs us validation loss on the MLP. docs/plan.md says what counts. I want an interval, not one number: dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout.

- Run `rb context --model <your model id>` first.
- Before any run: record the experiment, its settings from the configs, the plan's criterion cited to its line, and my criterion on the interval, with my words, needing 5 seeds. Freeze it.
- Run seeds 1 to 5 through `rb journal run`, and record each run.
- After each seed, tell me rb's finding as printed. Never round "point only" up to a result.

Finish with `rb show` of both criteria exactly as printed.

The repository trains a small MLP with no dropout and with dropout 0.1. docs/plan.md says: “Dropout 0.1 is acceptable if it costs at most 0.02 nats of validation loss.”

Two criteria read the same comparison, dropout’s validation loss minus no dropout’s. Each is listed by its id, its name in every command:

• mean-within-budget, the plan’s: the mean cost is at most 0.02 nats (--at-most 0.02);
• interval-within-budget, yours: the whole 95% interval stays within 0.02 nats (--noninferior 0.02), on at least 5 seeds (--min-n 5).

What you’ll see

After each seed, rb run add prints both findings. Seed by seed, as rb printed them:

Seeds	The reading, as rb prints it	mean-within-budget	interval-within-budget
1	+0.008 nats	estimate meets criterion · point only	criterion undetermined · point only
2	+0.0025 nats	estimate meets criterion · point only	criterion undetermined · point only
3	+0.006333 [-0.01508, +0.02775] nats	estimate meets criterion · unresolved	criterion undetermined · unresolved
4	+0.0055 [-0.006011, +0.01701] nats	criterion met · unresolved	criterion undetermined · unresolved
5	+0.0056 [-0.002184, +0.01338] nats	criterion met · unresolved	criterion met · unresolved

With one or two seeds there is no interval, and the plan’s criterion can only say estimate meets criterion · point only. At three seeds the interval reaches past the plan’s 0.02, so rb says “estimate meets criterion”, not “criterion met”. From four seeds the whole interval is inside 0.02. Your criterion stays undetermined until the fifth seed, because you declared that it needs five.

$ rb status
Dropout on a small MLP · dropout-on-a-small-mlp

Does dropout 0.1 cost us validation loss on the MLP? (open)
  On every drop criterion below: question link declared before the evidence (by production times); units: 5 seeds.
  The whole 95% interval of dropout's cost stays within 0.02 nats. change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb); unresolved: the interval includes 0; criterion met: the interval's worse bound +0.01338 inside 0.02; rule and question link declared before the evidence (by production times) · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout".
  Dropout 0.1 costs at most 0.02 nats of validation loss. change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb); unresolved: the interval includes 0; criterion met: estimate +0.0056 ≤ 0.02; rule and question link declared before the evidence (by production times).

Rules: 2 met
… (30 more lines)

Unresolved on the result line is not a failure: the interval still includes 0, so rb cannot say dropout changes the loss at all. Both criteria ask only that it not cost more than 0.02 nats, and it does not.

What rb did, step by step

1. Declare both criteria before any seed

   $ rb criterion add "Dropout 0.1 costs at most 0.02 nats of validation loss" -e drop -q q-drop --metric change.val_loss --at-most 0.02 --why docs/plan.md:3 --id mean-within-budget
   mean-within-budget: Dropout 0.1 costs at most 0.02 nats of validation loss
     change.val_loss ≤ 0.02 · primary · on drop
     declared; no evidence recorded yet
   
   Next: rb run add drop --from <the results file a run wrote> --set seed=<value>
   
   $ rb criterion add "The whole 95% interval of dropout's cost stays within 0.02 nats" -e drop -q q-drop --metric change.val_loss --noninferior 0.02 --min-n 5 --words "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout" --id interval-within-budget
   interval-within-budget: The whole 95% interval of dropout's cost stays within 0.02 nats
     change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5) · primary · on drop
     declared; no evidence recorded yet
   
   Next: rb run add drop --from <the results file a run wrote> --set seed=<value>

2. Freeze the experiment

   $ rb freeze drop -m "before the first seed"
   drop frozen: spec 12f84c212b8c · before the first seed
     drop: Dropout 0.1 vs none
       variants: none (baseline), p01 (candidate) · varies dropout
       criterion mean-within-budget (primary): change.val_loss ≤ 0.02
       criterion interval-within-budget (primary): change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5)   (words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout")
       optim.lr = 0.001 · verified · configs/train.yaml#optim.lr
       data.batch_size = 256 · verified · configs/train.yaml#data.batch_size
       seed = (each run) · per-run · no source
       none.dropout = 0.0 · verified · configs/none.yaml#dropout
       p01.dropout = 0.1 · verified · configs/p01.yaml#dropout
       look: mean-within-budget is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".
       look: interval-within-budget is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".
     Changing a setting or its source, a variant, what it varies, or a criterion now needs --amend --why "<reason>"; the amendment is kept with who made it.
   
   Next: rb run add drop --from <metrics.json> --command "..."

3. Run a seed, record it, read both criteria

   $ rb journal run -- python train.py --seed 1
   {"metrics": {"none.val_loss": 1.31, "p01.val_loss": 1.318}}
   exit 0. The command, its exit code and what it wrote are in the journal.
   It printed its metrics: none.val_loss=1.31, p01.val_loss=1.318. rb reconcile proposes the run of drop that records them.
   
   Since this session last looked:
     - out/seed1/ holds results no recorded experiment reads yet   (rb reconcile)
     - out/seed1/metrics.json: results landed and are not in the record   (rb reconcile)
   
   Next: rb reconcile

   $ rb run add drop --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"
   runy2bsxn on drop: none.val_loss=1.31, p01.val_loss=1.318 · seed=1
     Ran with the spec: 3 of its settings found in out/seed1/config.yaml, all matching.
     Produced: 2026-11-06 20:25:24 (file)
     mean-within-budget estimate meets criterion · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)
     interval-within-budget criterion undetermined · point only: change.val_loss: +0.008 nats · paired difference, 1 seed (rb)
     rb put the code in git at refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/3bf56b1d2d985fe410d77e394087a33e85d90a96 removes it.
   
   Next: rb show mean-within-budget  ·  rb show interval-within-budget
   
   $ rb run add drop --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"
   run5hv5cu on drop: none.val_loss=1.302, p01.val_loss=1.299 · seed=2
     Ran with the spec: 3 of its settings found in out/seed2/config.yaml, all matching.
     Produced: 2026-11-06 20:25:25 (file)
     mean-within-budget estimate meets criterion · point only: change.val_loss: +0.0025 nats · paired difference, 2 seeds (rb)
     interval-within-budget criterion undetermined · point only: change.val_loss: +0.0025 nats · paired difference, 2 seeds (rb)
   
   Next: rb show mean-within-budget  ·  rb show interval-within-budget
   
   $ rb run add drop --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"
   runaxeil6 on drop: none.val_loss=1.315, p01.val_loss=1.329 · seed=3
     Ran with the spec: 3 of its settings found in out/seed3/config.yaml, all matching.
     Produced: 2026-11-06 20:25:25 (file)
     mean-within-budget estimate meets criterion · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)
     interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.006333 [-0.01508, +0.02775] nats · paired difference, t interval, 3 seeds, 95% (rb)
   
   Next: rb show mean-within-budget  ·  rb show interval-within-budget
   
   $ rb run add drop --from out/seed4/metrics.json --set seed=4 --config out/seed4/config.yaml --command "python train.py --seed 4"
   runkeudey on drop: none.val_loss=1.308, p01.val_loss=1.311 · seed=4
     Ran with the spec: 3 of its settings found in out/seed4/config.yaml, all matching.
     Produced: 2026-11-06 20:25:26 (file)
     mean-within-budget criterion met · unresolved: change.val_loss: +0.0055 [-0.006011, +0.01701] nats · paired difference, t interval, 4 seeds, 95% (rb)
     interval-within-budget criterion undetermined · unresolved: change.val_loss: +0.0055 [-0.006011, +0.01701] nats · paired difference, t interval, 4 seeds, 95% (rb)
   
   Next: rb show mean-within-budget  ·  rb show interval-within-budget
   
   $ rb run add drop --from out/seed5/metrics.json --set seed=5 --config out/seed5/config.yaml --command "python train.py --seed 5"
   runkxmrdp on drop: none.val_loss=1.312, p01.val_loss=1.318 · seed=5
     Ran with the spec: 3 of its settings found in out/seed5/config.yaml, all matching.
     Produced: 2026-11-06 20:25:26 (file)
     mean-within-budget criterion met · unresolved: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)
     interval-within-budget criterion met · unresolved: change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)
   
   Next: rb show mean-within-budget  ·  rb show interval-within-budget

4. Read the criterion on its interval

   $ rb show interval-within-budget
   Criterion interval-within-budget on drop (primary, own): The whole 95% interval of dropout's cost stays within 0.02 nats · words (as given): "dropout passes only if the whole 95% interval stays within 0.02 nats of no dropout"
     interval-within-budget The whole 95% interval of dropout's cost stays within 0.02 nats
             criterion met · unresolved · change.val_loss no worse than 0.02, its interval's worse bound inside 0.02 (n ≥ 5)
             reads:     change.val_loss: +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)
             result:    unresolved: the interval includes 0
             rule:      criterion met: the interval's worse bound +0.01338 inside 0.02
             timing:    declared before the evidence (by production times)
             units:     5 seeds
             integrity: ok
     runy2bsxn: difference +0.008 (file) · unit seed=1 · seed=1
     run5hv5cu: difference -0.003 (file) · unit seed=2 · seed=2
     runaxeil6: difference +0.014 (file) · unit seed=3 · seed=3
     runkeudey: difference +0.003 (file) · unit seed=4 · seed=4
     runkxmrdp: difference +0.006 (file) · unit seed=5 · seed=5

5. Compare the arms

   rb compare gives the means; the criteria carry the intervals.

   $ rb compare drop
   drop  Dropout 0.1 vs none · varies dropout · 5 runs under the current spec · means over seeds
     variant  role       dropout  val_loss (nats)
     none     baseline   0.0      1.309 (n=5)
     p01      candidate  0.1      1.315 +0.0056 worse [primary] (n=5)
     the primary criteria read: val_loss p01 vs none (mean-within-budget reads +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)), val_loss p01 vs none (interval-within-budget reads +0.0056 [-0.002184, +0.01338] nats · paired difference, t interval, 5 seeds, 95% (rb)); deltas are means, the findings carry the intervals (rb show <criterion>)

Why it matters

“It costs 0.008 nats” from one seed and “it costs between -0.002 and +0.013” from five are different claims. rb says which one you have.