Declare the criterion before the runs

This cookbook declares a criterion and freezes it before the first run, then shows what rb says of one declared after.

A threshold picked after the number came in is not a test: it was fitted to the result. Declare the criterion and lock it before the first run, and rb’s finding says so. A criterion declared later says that too.

Paste this into your agent

Declare the criterion and freeze it before any run, then run the seeds.

Use Rabbit Brain to find out if warmup is worth it for the 6-layer GPT. docs/plan.md says what counts.

- Run `rb context --model <your model id>` first.
- Before any run: record the question, the experiment with both arms and every setting cited to its config, and the criterion docs/plan.md states, cited to its line. Then freeze the experiment.
- Then run seeds 1 to 3 through `rb journal run -- python train.py --seed <N>`, and record each run from its output file, with its config.
- Compute nothing yourself.

Finish with `rb show` of the criterion exactly as printed.

The repository: a 6-layer GPT trained with and without learning-rate warmup, configs/ for each arm, and docs/plan.md: “Warmup ships if it lowers final validation loss by at least 0.02 nats against no warmup.”

What you’ll see

The plan’s criterion, read after three seeds:

$ rb show warmup-helps
Criterion warmup-helps on warmup (primary, own): Warmup lowers final validation loss by at least 0.02 nats
  warmup-helps Warmup lowers final validation loss by at least 0.02 nats
          criterion met · gain · change.val_loss ≤ -0.02 (n ≥ 3)
          reads:     change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb)
          result:    gain: the interval clear of 0 on the better side (lower is better)
          rule:      criterion met: estimate -0.02767 ≤ -0.02
          timing:    declared before the evidence (by production times)
          units:     3 seeds
          integrity: ok
  rungpbpyp: difference -0.031 (file) · unit seed=1 · seed=1
  runcryu4l: difference -0.026 (file) · unit seed=2 · seed=2
  run7oke3c: difference -0.026 (file) · unit seed=3 · seed=3

warmup-helps is the criterion’s id: its name in every command. Its timing reads declared before the evidence: the criterion was declared, and the experiment frozen, before any seed ran.

Now suppose someone raises the bar to 0.03 nats after seeing these numbers. The experiment is frozen, so rb refuses the change until it gives its reason:

$ rb criterion add "Warmup lowers final validation loss by at least 0.03 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.03 --min-n 3 --id stricter-warmup-bar
rb criterion add: Experiment warmup was frozen at 2026-11-06T20:25:15.509+00:00 (e84183e9a687); this changes it (new criterion stricter-warmup-bar: change.val_loss ≤ -0.03 (n ≥ 3)).
  → Give the change its reason: the same command with --amend --why "<reason>". The amendment is recorded, by whoever makes it.

$ rb criterion add "Warmup lowers final validation loss by at least 0.03 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.03 --min-n 3 --id stricter-warmup-bar --amend --why "a stricter bar, set after seeing the runs"
stricter-warmup-bar: Warmup lowers final validation loss by at least 0.03 nats
  change.val_loss ≤ -0.03 (n ≥ 3) · primary · on warmup
  declared after the evidence existed: all 3 units were produced before it was declared
  it already reads change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb): criterion not met: estimate -0.02767 > -0.03

Next: rb run add warmup --from <the results file a run wrote> --set seed=<value>

Both criteria read the same three runs. rb status keeps them apart by when each was declared:

$ rb status
Warmup on a small GPT · warmup-on-a-small-gpt

Is warmup worth it for the 6-layer GPT? (open)
  On every warmup criterion below: units: 3 seeds.
  Warmup lowers final validation loss by at least 0.03 nats. change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (lower is better); criterion not met: estimate -0.02767 > -0.03; rule and question link declared after the evidence existed: all 3 units were produced before it was declared. the interval [-0.03484, -0.0205] crosses -0.03: missed on the estimate, not across the whole interval.
  Warmup lowers final validation loss by at least 0.02 nats. change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (lower is better); criterion met: estimate -0.02767 ≤ -0.02; rule and question link declared before the evidence (by production times).

Rules: 1 met · 1 not met

Needs judgment (1)
  - stricter-warmup-bar misses its rule and no call on it is recorded: Warmup lowers final validation loss by at least 0.03 nats
      rb decide stricter-warmup-bar accept|reject|investigate --why "..."

The design of 1 experiment is not confirmed: rb confirm lists it.

Can be done now (1)
  - warmup: 1 criterion was declared after its evidence existed: a reading of its own needs its subject's runs made after the declaration
      rb run add warmup --from <the results file a run wrote> --set seed=<value>   (run after the declaration)

rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-06 20:25 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.

Rules
  warmup-helps Warmup lowers final validation loss by at least 0.02 nats
          criterion met · gain · change.val_loss ≤ -0.02 (n ≥ 3)
… (21 more lines)

The late criterion reads declared after the evidence existed, and its miss waits for a call under Needs judgment. Calls shows that call being made.

What rb did, step by step

Record the experiment and cite its settings

$ rb experiment add "Warmup vs no warmup" --id warmup -q q-warmup --baseline none --candidate warmup --varies warmup_steps
Experiment warmup: Warmup vs no warmup · none (baseline) vs warmup (candidate) · varies warmup_steps · under q-warmup
  Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
  Made 2 variants: none (baseline), warmup (candidate); a setting of one is rb spec set warmup none.<setting> <value>.
  warmup declares it varies warmup_steps, but no variant has a value for it yet; give each variant its own: rb spec set warmup <variant>.warmup_steps <value> --source <file#key>

Next: rb spec set warmup --from <config.yaml> --keys "<the keys that define it>"  ·  rb criterion add "<what it should show>" -e warmup -q q-warmup --metric <name> --at-most <x>

$ rb spec set warmup --from configs/base.yaml --keys "optim.*,data.*"
warmup: 4 settings from configs/base.yaml @90e2f71 · 4 verified

$ rb spec set warmup none.warmup_steps 0 --source "configs/no-warmup.yaml#warmup_steps"
warmup/none.warmup_steps = 0 · verified in configs/no-warmup.yaml#warmup_steps

$ rb spec set warmup warmup.warmup_steps 500 --source "configs/warmup.yaml#warmup_steps"
warmup/warmup.warmup_steps = 500 · verified in configs/warmup.yaml#warmup_steps

$ rb spec set warmup seed --per-run
warmup/seed · per-run: each run gives its own, with --set seed=<value> on rb run add

Declare the plan’s criterion

--why "docs/plan.md:3": rb checks that the line is there and states the criterion’s threshold.

$ rb criterion add "Warmup lowers final validation loss by at least 0.02 nats" -e warmup -q q-warmup --metric change.val_loss --at-most -0.02 --min-n 3 --why docs/plan.md:3 --id warmup-helps
warmup-helps: Warmup lowers final validation loss by at least 0.02 nats
  change.val_loss ≤ -0.02 (n ≥ 3) · primary · on warmup
  declared; no evidence recorded yet

Next: rb run add warmup --from <the results file a run wrote> --set seed=<value>

Freeze it

The freeze records a hash of the variants, settings and criteria, and prints each criterion it locks.

$ rb freeze warmup -m "before the runs"
warmup frozen: spec e84183e9a687 · before the runs
  warmup: Warmup vs no warmup
    variants: none (baseline), warmup (candidate) · varies warmup_steps
    criterion warmup-helps (primary): change.val_loss ≤ -0.02 (n ≥ 3)
    optim.lr = 0.0006 · verified · configs/base.yaml#optim.lr
    optim.weight_decay = 0.1 · verified · configs/base.yaml#optim.weight_decay
    data.batch_size = 64 · verified · configs/base.yaml#data.batch_size
    data.seq_len = 256 · verified · configs/base.yaml#data.seq_len
    seed = (each run) · per-run · no source
    none.warmup_steps = 0 · verified · configs/no-warmup.yaml#warmup_steps
    warmup.warmup_steps = 500 · verified · configs/warmup.yaml#warmup_steps
    look: warmup-helps is locked as the test now. Check it is the one you mean: a change to it needs --amend --why "<reason>".
  Changing a setting or its source, a variant, what it varies, or a criterion now needs --amend --why "<reason>"; the amendment is kept with who made it.

Next: rb run add warmup --from <metrics.json> --command "..."

Run each seed through rb journal run

The command, its exit code and the files it wrote are kept, even if nothing else is written down.

$ rb journal run -- python train.py --seed 1
seed 1: val_loss none 2.412 warmup 2.381
{"metrics": {"none.val_loss": 2.412, "warmup.val_loss": 2.381}}
exit 0. The command, its exit code and what it wrote are in the journal.
It printed its metrics: none.val_loss=2.412, warmup.val_loss=2.381. rb reconcile proposes the run of warmup that records them.

Since this session last looked:
  - out/seed1/ holds results no recorded experiment reads yet   (rb reconcile)
  - out/seed1/metrics.json: results landed and are not in the record   (rb reconcile)

Next: rb reconcile

Record each run from its output

$ rb run add warmup --from out/seed1/metrics.json --set seed=1 --config out/seed1/config.yaml --command "python train.py --seed 1"
rungpbpyp on warmup: none.val_loss=2.412, warmup.val_loss=2.381 · seed=1
  Ran with the spec: 5 of its settings found in out/seed1/config.yaml, all matching.
  Produced: 2026-11-06 20:25:15 (file)
  warmup-helps criterion undetermined · point only: change.val_loss: -0.031 nats · paired difference, 1 seed (rb)
  rb put the code in git at refs/rb/snapshots/a811c430d496facb6736463d66d13e9744b1ca54, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/a811c430d496facb6736463d66d13e9744b1ca54 removes it.

Next: rb show warmup-helps

$ rb run add warmup --from out/seed2/metrics.json --set seed=2 --config out/seed2/config.yaml --command "python train.py --seed 2"
runcryu4l on warmup: none.val_loss=2.405, warmup.val_loss=2.379 · seed=2
  Ran with the spec: 5 of its settings found in out/seed2/config.yaml, all matching.
  Produced: 2026-11-06 20:25:16 (file)
  warmup-helps criterion undetermined · point only: change.val_loss: -0.0285 nats · paired difference, 2 seeds (rb)

Next: rb show warmup-helps

$ rb run add warmup --from out/seed3/metrics.json --set seed=3 --config out/seed3/config.yaml --command "python train.py --seed 3"
run7oke3c on warmup: none.val_loss=2.418, warmup.val_loss=2.392 · seed=3
  Ran with the spec: 5 of its settings found in out/seed3/config.yaml, all matching.
  Produced: 2026-11-06 20:25:16 (file)
  warmup-helps criterion met · gain: change.val_loss: -0.02767 [-0.03484, -0.0205] nats · paired difference, t interval, 3 seeds, 95% (rb)

Next: rb show warmup-helps

Why it matters

A criterion declared first is a test. rb records which came first, so a reader never has to take it on trust.

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0