Record a folder per run

This cookbook records a repository where each run keeps its own folder, with its config and its metrics.

Your eval script writes results/<run>/config.yaml and results/<run>/metrics.json for every run: the command, the commit, whether the code had uncommitted changes, and the numbers. You want each folder recorded as the run it is, with what its config says.

Paste this into your agent

Record each run folder in results/ as a run, from what its own config says.

Each folder in results/ is one run of eval.py. Record them with Rabbit Brain.

- Run `rb context --model <your model id>` first, then `rb reconcile`.
- Record the question and the experiment, with each setting cited to the config that states it, and the metric.
- Run `rb reconcile` again, and record each folder with the `rb run add` line it prints, word for word.
- Compute nothing yourself: rb computes the findings.

Finish with `rb reconcile` and `rb status` exactly as printed.

The repository scores a small model on 500 grade-school maths problems with a plain prompt and with three worked examples, two seeds each. Each run has its folder in results/, r01 to r04. The fourth one’s config and metrics:

$ cat results/r04/config.yaml
command: python eval.py --config configs/examples.yaml --seed 2 --out results/r04
commit: 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc
dirty: true
started_at: 2026-10-28T11:30:00Z
prompt: examples
problems: 500
seed: 2

$ cat results/r04/metrics.json
{
  "accuracy": 46.9,
  "finished_at": "2026-10-28T11:53:00Z"
}

The fourth run started while prompts/examples.txt held an edit nobody committed, so its config says dirty: true.

What you’ll see

rb status in your own terminal, once the agent is done:

$ rb status
Worked examples in the prompt · worked-examples-in-the-prompt
Kept with rb since 2026-11-06. All 4 runs were recorded afterwards, not as they happened.

Do three worked examples in the prompt raise accuracy on grade-school maths? (open)
  No criterion is linked directly to this question.
  Worked examples vs a plain prompt · open

Rules: none yet

The design of 1 experiment is not confirmed: rb confirm lists it.

Facts that ask for nothing (1)
  - examples: runs report examples.accuracy, plain.accuracy, which no recorded criterion reads

rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-06 20:25 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.

Experiments
  examples Worked examples vs a plain prompt · open · plain (baseline) vs examples (candidate) · varies prompt · not frozen · settings 3 verified · 0 provisional · 0 unknown · 1 per-run · runs 4
          to run again: code partial · environment unknown · files ok

Runs: 4 recorded on 1 experiment · every receipt is what rb wrote

Next: rb confirm

The four runs are recorded, and the line under the title says they were recorded afterwards, not as they happened.
“environment unknown”: these runs were made before rb was watching, so their receipts cannot say which machine made them. Each receipt keeps the commit and the command its config gives.
No criterion is recorded yet: no file here states what counts. “runs report examples.accuracy, plain.accuracy, which no recorded criterion reads” says so.

What rb did, step by step

Read what the repository already says

A results folder that holds a run’s own config beside a results file is a folder of one run. For each, rb reconcile says what its config says of the run, as the file’s words with their lines: the command, the commit, dirty, and when it started. It names the settings that differ between the folders, and prints an rb run add line for each. With no experiment yet, the line reads <experiment>. It writes nothing.

$ rb reconcile
The repository at 855f30c: 4 results folders.

4 results folders hold results files no recorded run named, in results/ (4 folders). With no report of its own none is an experiment: rb run add <experiment> --from <its file> records a run from one.

Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:
  results/r01/  config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)
    rb run add '<experiment>' --from results/r01/metrics.json --config results/r01/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' --command-source results/r01/config.yaml:1
  results/r02/  config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)
    rb run add '<experiment>' --from results/r02/metrics.json --config results/r02/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' --command-source results/r02/config.yaml:1
  results/r03/  config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)
    rb run add '<experiment>' --from results/r03/metrics.json --config results/r03/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' --command-source results/r03/config.yaml:1
  results/r04/  config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)
    results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran
    rb run add '<experiment>' --from results/r04/metrics.json --config results/r04/config.yaml --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' --command-source results/r04/config.yaml:1
  Settings that differ between them (2):
    prompt: plain (r01, r03), examples (r02, r04)
    seed: 1 (r01, r02), 2 (r03, r04)

Record the question, the experiment and its settings

$ rb question add "Do three worked examples in the prompt raise accuracy on grade-school maths?" --id q-examples
q-examples: Do three worked examples in the prompt raise accuracy on grade-school maths?

Next: rb experiment add "<what would answer it>" -q q-examples --hypothesis "<what you expect>"

$ rb experiment add "Worked examples vs a plain prompt" --id examples -q q-examples --baseline plain --candidate examples --varies prompt
Experiment examples: Worked examples vs a plain prompt · plain (baseline) vs examples (candidate) · varies prompt · under q-examples
  Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
  Made 2 variants: plain (baseline), examples (candidate); a setting of one is rb spec set examples plain.<setting> <value>.
  examples declares it varies prompt, but no variant has a value for it yet; give each variant its own: rb spec set examples <variant>.prompt <value> --source <file#key>

Next: rb spec set examples --from <config.yaml> --keys "<the keys that define it>"  ·  rb criterion add "<what it should show>" -e examples -q q-examples --metric <name> --at-most <x>

$ rb spec set examples plain.prompt plain --source "configs/plain.yaml#prompt"
examples/plain.prompt = plain · verified in configs/plain.yaml#prompt

$ rb spec set examples examples.prompt examples --source "configs/examples.yaml#prompt"
examples/examples.prompt = examples · verified in configs/examples.yaml#prompt

$ rb spec set examples problems 500 --source "configs/plain.yaml#problems"
examples/problems = 500 · verified in configs/plain.yaml#problems

$ rb spec set examples seed --per-run
examples/seed · per-run: each run gives its own, with --set seed=<value> on rb run add

$ rb metric add accuracy --unit % --maximize
Metric accuracy (%): higher is better

Read the folders again

Now each line names the experiment, the variant the config’s prompt names, the metric, and the seed:

$ rb reconcile
The repository at 855f30c: 4 results folders.

4 results folders hold results files no recorded run named, in results/ (4 folders). With no report of its own none is an experiment: rb run add <experiment> --from <its file> records a run from one.

Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:
  results/r01/  config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)
    rb run add examples --from results/r01/metrics.json --variant plain --keys accuracy --config results/r01/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' --command-source results/r01/config.yaml:1
  results/r02/  config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)
    rb run add examples --from results/r02/metrics.json --variant examples --keys accuracy --config results/r02/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' --command-source results/r02/config.yaml:1
  results/r03/  config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)
    rb run add examples --from results/r03/metrics.json --variant plain --keys accuracy --config results/r03/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' --command-source results/r03/config.yaml:1
  results/r04/  config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)
    results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran
    rb run add examples --from results/r04/metrics.json --variant examples --keys accuracy --config results/r04/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' --command-source results/r04/config.yaml:1
  Settings that differ between them (2):
    prompt: plain (r01, r03), examples (r02, r04)
    seed: 1 (r01, r02), 2 (r03, r04)

Record each folder with its line

Each run is checked against its own config (“Ran with the spec: 3 of its settings found in results/r01/config.yaml, all matching”), and is dated by the finished_at its metrics file gives. r04’s receipt keeps what its config says: “dirty: true: the code had uncommitted changes when it ran”.

$ rb run add examples --from results/r01/metrics.json --variant plain --keys accuracy --config results/r01/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:20:00Z --command "python eval.py --config configs/plain.yaml --seed 1 --out results/r01" --command-source results/r01/config.yaml:1
runlfb7wl on examples: plain.accuracy=41.2 · seed=1
  Ran with the spec: 3 of its settings found in results/r01/config.yaml, all matching.
  Produced: 2026-10-28 10:20:00 (reported)
  rb put the code in git at refs/rb/snapshots/6edfbdd7cd3fe9b2350eee1cfbc31b16fd6806d1, so it outlives a deleted branch or worktree; git update-ref -d refs/rb/snapshots/6edfbdd7cd3fe9b2350eee1cfbc31b16fd6806d1 removes it.

Since this session last looked:
  - 3 results folders hold results written before the record began, which no recorded experiment reads yet: results/r02/, results/r03/, results/r04/   (rb reconcile)

Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric plain.accuracy --at-most <x>   (write the criterion before the next run)

$ rb run add examples --from results/r02/metrics.json --variant examples --keys accuracy --config results/r02/config.yaml --set seed=1 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T10:52:00Z --command "python eval.py --config configs/examples.yaml --seed 1 --out results/r02" --command-source results/r02/config.yaml:1
runlsdwkq on examples: examples.accuracy=47.8 · seed=1
  Ran with the spec: 3 of its settings found in results/r02/config.yaml, all matching.
  Produced: 2026-10-28 10:52:00 (reported)

Since this session last looked:
  - 2 results folders hold results written before the record began, which no recorded experiment reads yet: results/r03/, results/r04/   (rb reconcile)

Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric examples.accuracy --at-most <x>   (write the criterion before the next run)

$ rb run add examples --from results/r03/metrics.json --variant plain --keys accuracy --config results/r03/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:21:00Z --command "python eval.py --config configs/plain.yaml --seed 2 --out results/r03" --command-source results/r03/config.yaml:1
runo6wdvc on examples: plain.accuracy=40.6 · seed=2
  Ran with the spec: 3 of its settings found in results/r03/config.yaml, all matching.
  Produced: 2026-10-28 11:21:00 (reported)

Since this session last looked:
  - results/r04/ holds results written before the record began, which no recorded experiment reads yet   (rb reconcile)

Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric plain.accuracy --at-most <x>   (write the criterion before the next run)

$ rb run add examples --from results/r04/metrics.json --variant examples --keys accuracy --config results/r04/config.yaml --set seed=2 --commit 48917c6e0ef4fc881eff8e5fc999ce9aabe62abc --produced-at 2026-10-28T11:53:00Z --command "python eval.py --config configs/examples.yaml --seed 2 --out results/r04" --command-source results/r04/config.yaml:1
runey4xee on examples: examples.accuracy=46.9 · seed=2
  Ran with the spec: 3 of its settings found in results/r04/config.yaml, all matching.
  results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran.
  Produced: 2026-10-28 11:53:00 (reported)

Next: rb criterion add "<what it should show>" -e examples -q q-examples --metric examples.accuracy --at-most <x>   (write the criterion before the next run)

Read the repository again

Each folder now says which run it was recorded as, and that the runs of examples read it:

$ rb reconcile
The repository at 855f30c: 4 results folders.

Run folders, each with its own record of the run (4). What each record says is the file's, not what rb saw:
  results/r01/  config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 1 --out results/r01' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:00:00Z (line 4)
    recorded as runlfb7wl: rb show runlfb7wl
  results/r02/  config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 1 --out results/r02' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T10:30:00Z (line 4)
    recorded as runlsdwkq: rb show runlsdwkq
  results/r03/  config.yaml says command: 'python eval.py --config configs/plain.yaml --seed 2 --out results/r03' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: false (line 3) · started_at: 2026-10-28T11:00:00Z (line 4)
    recorded as runo6wdvc: rb show runo6wdvc
  results/r04/  config.yaml says command: 'python eval.py --config configs/examples.yaml --seed 2 --out results/r04' (line 1) · commit: 48917c6e0ef4 (line 2) · dirty: true (line 3) · started_at: 2026-10-28T11:30:00Z (line 4)
    results/r04/config.yaml:3 says dirty: true: the code had uncommitted changes when it ran
    recorded as runey4xee: rb show runey4xee
  Settings that differ between them (2):
    prompt: plain (r01, r03), examples (r02, r04)
    seed: 1 (r01, r02), 2 (r03, r04)

results/r01/  read by the runs of examples
  files: 2 results files
  git: both report and results files are tracked · folder unchanged since the last commit

results/r02/  read by the runs of examples
  files: 2 results files
  git: both report and results files are tracked · folder unchanged since the last commit

results/r03/  read by the runs of examples
  files: 2 results files
  git: both report and results files are tracked · folder unchanged since the last commit

results/r04/  read by the runs of examples
  files: 2 results files
  git: both report and results files are tracked · folder unchanged since the last commit

Why it matters

A run’s own folder already says how it ran. rb records it from those words, with their lines, and says where the config reports uncommitted code.

Rabbit Brainrabbitbrain.ai · GitHub · PyPI · Privacy · Apache-2.0