Check a summary's claims against the runs

This cookbook checks each claim in an agent’s summary against the runs it is about.

An agent’s session ended with a confident summary, and you are about to repeat it to your team. Before you do, each claim should be checked against the runs, by criteria rb reads, not by another summary.

Paste this into your agent

Paste this into your agentCopy

Check every claim in SUMMARY.md against the runs, with rb doing the reading.

My last session wrote SUMMARY.md. Check each claim in it against the runs with Rabbit Brain, and take nothing on the summary's word.

- Run `rb context --model <your model id>` first.
- Record the experiment, and every run from its own file in `results/`.
- Turn each claim into the criterion it states, cited to its line: `--source SUMMARY.md --quote "<the claim's words>"`. Use the claim's own number, with no threshold of yours.
- A claim no criterion can read goes in a note that names the files you read.
- Compute nothing yourself: rb computes the findings.

Finish with `rb status` exactly as printed, and for each claim, what rb says of it.

We ran it on the INT8 ResNet-20 repository, with three seeds in results/int8-ptq/ and this SUMMARY.md from an earlier session:

SUMMARY.md

# Session summary

- INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed.
- INT8 averages 91.62 top-1 over the three seeds.
- 512 calibration images are enough.

What you’ll see

$ rb status
INT8 ResNet-20 on CIFAR-10 · int8-resnet-20-on-cifar-10
Kept with rb since 2026-11-06. All 3 runs were recorded afterwards, not as they happened.

Does INT8 match FP32 top-1 on CIFAR-10? (open)
  On every int8-ptq criterion below: question link declared after the evidence existed: all 3 units were produced before it was declared; units: 3 seeds.
  The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here. top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb); gain: the interval clear of 0 on the better side (higher is better); not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2); rule and question link declared after the evidence existed: all 3 units were produced before it was declared.
  The summary's claim that INT8 averages 91.62 top-1 is checked here. int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb); measured: int8 on its own, with its interval, so neither better nor worse; reproduced on the estimate: estimate 91.6233 within 91.62 ± 0.005; rule and question link declared after the evidence existed: all 3 units were produced before it was declared. the interval [91.48, 91.77] crosses 91.615 and 91.625: reproduced on the estimate, not across the whole interval.
  experiment context: int8-ptq · INT8 post-training quantization
          note n64w2hj: No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/. · not confirmed

Rules: 1 reproduced, on the estimate alone · 1 not reproduced

Needs judgment (1)
  - every-seed-claim is not reproduced and no call on it is recorded: The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
      rb decide every-seed-claim accept|reject|investigate --why "..."

The design of 1 experiment is not confirmed: rb confirm lists it.

Facts that ask for nothing (1)
  - int8-ptq: 2 criteria were declared after their evidence existed

rb looks in out/, outputs/, results/, eval-out/ and metrics/ at the top, in folders named for results (results/, eval/, metrics/, reports/, artifacts/ and the like), down to 4 levels from the top, at the files in the repository's top folder, and at .json, .yaml and .toml files git shows as new or changed. It last looked at 2026-11-06 20:25 UTC. It reads what code, configs, docs and database rows say only where the record points to them, and it sees only the commands run through rb journal run.

Rules
  every-seed-claim The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
          not reproduced · gain · each unit's top1: fp32 vs int8 ≤ 0.05
          reads:     top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb)
          result:    gain: the interval clear of 0 on the better side (higher is better)
          rule:      not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2)
          timing:    declared after the evidence existed: all 3 units were produced before it was declared
          units:     3 seeds
  int8-mean-claim The summary's claim that INT8 averages 91.62 top-1 is checked here
          reproduced on the estimate · measured · int8.top1 = 91.62 ± 0.005
          reads:     int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb)
          result:    measured: int8 on its own, with its interval, so neither better nor worse
          rule:      reproduced on the estimate: estimate 91.6233 within 91.62 ± 0.005
          timing:    declared after the evidence existed: all 3 units were produced before it was declared
          units:     3 seeds
          · the interval [91.48, 91.77] crosses 91.615 and 91.625: reproduced on the estimate, not across the whole interval

Experiments
  int8-ptq INT8 post-training quantization · open · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · not frozen · settings 3 verified · 0 provisional · 0 unknown · 1 per-run · runs 3
          to run again: code unknown · environment unknown · files ok

Runs: 3 recorded on 1 experiment · every receipt is what rb wrote

Next: rb confirm  ·  rb decide every-seed-claim accept|reject|investigate --why "..."

every-seed-claim and int8-mean-claim are the criteria’s ids: their names in every command.

Claim by claim:

• “It never loses more than 0.05 points of top-1 on any seed”: not reproduced. rb names the seeds that break it: seed 1 and seed 2. This criterion compares FP32 against INT8, so “gain” on its result line means FP32 is ahead.
• “INT8 averages 91.62 top-1”: reproduced on the estimate. The mean over three seeds is 91.6233, within 0.005 of 91.62. The interval is wider than that: reproduced on the estimate, not across the whole interval. The tally says so too: “1 reproduced, on the estimate alone”.
• “512 calibration images are enough”: no criterion reads it, since no run compares calibration sizes. The agent left a note that says so, with the files it read.

The miss waits for you under Needs judgment: whether to accept, reject or investigate it is your call, not the agent’s.

What rb did, step by step

1. Record the experiment and its settings

   $ rb experiment add "INT8 post-training quantization" --id int8-ptq -q q-int8 --baseline fp32 --candidate int8 --varies precision,calib_images
   Experiment int8-ptq: INT8 post-training quantization · fp32 (baseline) vs int8 (candidate) · varies precision, calib_images · under q-int8
     Planned: nothing in the record tests it yet, and it shows as a planned direction until its first run or analysis.
     Made 2 variants: fp32 (baseline), int8 (candidate); a setting of one is rb spec set int8-ptq fp32.<setting> <value>.
     int8-ptq declares it varies precision, but no variant has a value for it yet; give each variant its own: rb spec set int8-ptq <variant>.precision <value> --source <file#key>
     int8-ptq declares it varies calib_images, but no variant has a value for it yet; give each variant its own: rb spec set int8-ptq <variant>.calib_images <value> --source <file#key>
   
   Next: rb spec set int8-ptq --from <config.yaml> --keys "<the keys that define it>"  ·  rb criterion add "<what it should show>" -e int8-ptq -q q-int8 --metric <name> --at-most <x>

   $ rb spec set int8-ptq fp32.precision fp32 --source "configs/fp32.yaml#precision"
   int8-ptq/fp32.precision = fp32 · verified in configs/fp32.yaml#precision
   
   $ rb spec set int8-ptq int8.precision int8 --source "configs/int8.yaml#precision"
   int8-ptq/int8.precision = int8 · verified in configs/int8.yaml#precision
   
   $ rb spec set int8-ptq int8.calib_images 512 --source "configs/int8.yaml#calib_images"
   int8-ptq/int8.calib_images = 512 · verified in configs/int8.yaml#calib_images
   
   $ rb spec set int8-ptq seed --per-run
   int8-ptq/seed · per-run: each run gives its own, with --set seed=<value> on rb run add

2. Turn each claim into the criterion it states

   A criterion with --source must find its number in the quoted line, and rb checks the line is there. The “never loses more than 0.05” claim is written as FP32 against INT8, at most 0.05 on every seed (--each), so the number in the criterion is the number in the line.

   $ rb criterion add "The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here" -e int8-ptq -q q-int8 --metric top1 --subject fp32 --reference int8 --at-most 0.05 --each --source SUMMARY.md --quote "it never loses more than 0.05 points of top-1 on any seed" --id every-seed-claim
   every-seed-claim: The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
     each unit's top1: fp32 vs int8 ≤ 0.05 · primary · on int8-ptq
     cited: SUMMARY.md "it never loses more than 0.05 points of top-1 on any seed" · states 0.05 (read '- INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed.')
     declared; no evidence recorded yet
   
   Next: rb run add int8-ptq --from <the results file a run wrote> --set seed=<value>
   
   $ rb criterion add "The summary's claim that INT8 averages 91.62 top-1 is checked here" -e int8-ptq -q q-int8 --metric int8.top1 --equals 91.62 --tolerance 0.005 --source SUMMARY.md --quote "INT8 averages 91.62 top-1 over the three seeds" --id int8-mean-claim
   int8-mean-claim: The summary's claim that INT8 averages 91.62 top-1 is checked here
     int8.top1 = 91.62 ± 0.005 · primary · on int8-ptq
     cited: SUMMARY.md "INT8 averages 91.62 top-1 over the three seeds" · states 91.62 (read '- INT8 averages 91.62 top-1 over the three seeds.')
     declared; no evidence recorded yet
   
   Next: rb run add int8-ptq --from <the results file a run wrote> --set seed=<value>

3. Record each run from its own file

   The findings change as each run lands. After the third:

   $ rb run add int8-ptq --from results/int8-ptq/seed3/metrics.json --set seed=3 --config results/int8-ptq/seed3/config.yaml
   rundacsx5 on int8-ptq: fp32.top1=91.74, int8.top1=91.69 · seed=3
     Ran with the spec: 1 of its settings found in results/int8-ptq/seed3/config.yaml, matching.
     Produced: 2026-10-20 10:00:00 (file)
     every-seed-claim not reproduced · gain: top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb)
     int8-mean-claim reproduced on the estimate · measured: int8.top1: 91.6233 [91.4778, 91.7689] % · mean, t interval, 3 seeds, 95% (rb)
   
   Next: rb show every-seed-claim  ·  rb show int8-mean-claim

4. Note the claim no criterion can read

   $ rb note add "No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/." -e int8-ptq
   Note n64w2hj on int8-ptq: No file I read compares calibration sizes, so no rule reads the claim that 512 calibration images are enough (SUMMARY.md:5). Files read: SUMMARY.md, configs/int8.yaml, results/int8-ptq/seed1/, seed2/, seed3/.
     Kept verbatim with who wrote it; a note changes no finding.

5. Read each claim’s criterion

   $ rb show every-seed-claim
   Criterion every-seed-claim on int8-ptq (primary, cited): The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
     cited: SUMMARY.md "it never loses more than 0.05 points of top-1 on any seed" · states 0.05, read '- INT8 matches FP32: it never loses more than 0.05 points of top-1 on any seed.'
     every-seed-claim The summary's claim that INT8 never loses more than 0.05 points of top-1 on any seed is checked here
             not reproduced · gain · each unit's top1: fp32 vs int8 ≤ 0.05
             reads:     top1: fp32 vs int8: +0.08 [+0.005476, +0.1545] % · paired difference, t interval, 3 seeds, 95% (rb)
             result:    gain: the interval clear of 0 on the better side (higher is better)
             rule:      not reproduced: 2 of 3 units are above 0.05 (seed=1, seed=2)
             timing:    declared after the evidence existed: all 3 units were produced before it was declared
             units:     3 seeds
             integrity: ok
     run4nbkcp: difference +0.11 (file) · unit seed=1 · produced before the declaration · seed=1
     run7wjy3b: difference +0.08 (file) · unit seed=2 · produced before the declaration · seed=2
     rundacsx5: difference +0.05 (file) · unit seed=3 · produced before the declaration · seed=3

Why it matters

A summary is a claim about the runs. rb reads the runs, and says which claims hold, which do not, and which nothing checked.