Rabbit Brain

Install

pip install rabbit-brain

Python 3.10 or later. The core has one dependency. Add [raft] for the built-in RAFT adapter and torch, or [evidence] for rendered evidence sheets with your own adapter.

pip install "rabbit-brain[raft]"

Your first comparison, with no setup

Nothing here touches your model. It builds a run from the bundled example so you can see the shape of the output.

rb example
rb findings <run> --top 5
rb case <run> <case> --render

Every command takes --json. rb runs lists what you have. Or look at a real 200-case comparison before installing anything.

Your own model

Two ways in. Import numbers your evaluator already produces:

rb import results.json      # or a CSV: rb import results.csv --project ...
rb schema example           # the format, with every field

Or let Rabbit Brain run both checkpoints, which is the only way to get trajectories and rendered evidence, because only a run sees inside the model:

rb init --project my-flow --adapter raft --model-code ./raft --dataset ./data
rb doctor && rb verify-hook --checkpoint ckpt/current.pth
rb verify-adapter           # proves our numbers match your own evaluator
rb run --baseline ckpt/current.pth --candidate ckpt/candidate.pth

rb verify-adapter is the one worth understanding. It runs your model through its own loader, forward call and metric formula and compares. A disagreement beyond 0.001 relative stops the run and writes nothing, because a harness that scores slightly differently produces plausible numbers that are not about your model. Not sure it fits? Describe your setup and you get the configuration back.

In CI

Keep the cases you care about, then fail the build when one of them breaks.

rb check save <run> <case>          # keeps a limit for this case
rb check run <run> --fail-on checks # exit 1 when a saved check fails

The exit code is also non-zero when your policy declared a limit this version cannot evaluate, so a required measurement that was never taken does not quietly pass. A working GitHub Actions workflow is in the repository .

A red job is not a blocked merge. Someone with repository admin has to add the job to branch protection as a required status check before the gate enforces anything.

Commands

All 19 of them, read from the installed package.

rb case

one case: numbers, trajectory statistics, reasoning

rb check

saved checks: save, run, list, rm

• rb check save
• rb check run
• rb check list
• rb check rm

rb docs

print AGENTS.md (or --errors for the error table)

rb doctor

check the environment, adapter, model code, dataset and checkpoints

rb example

create a run from the built-in example data, or write it as a version-1 JSON

rb findings

the ranked queue, summary and verdict for a run

rb import

import a version-1 results JSON or a metrics CSV into a new run

rb init

write rb.toml for this project (or --demo for a synthetic project that runs anywhere)

rb onboard

turn a described setup into a configured project, or into the honest shape of one

rb plans

what a workspace costs (no token needed)

rb report

write report.md for a run (the receipt)

rb run

evaluate the current and candidate checkpoints on the case set and write a run

rb runs

list runs

rb schema

print a JSON Schema, the minimal example, or the full example

rb share

write the run's anonymised statistics to share.json for the calibration corpus; nothing is sent

rb verify-adapter

compare the adapter's per-case error with the model repository's own evaluation on a few cases

rb verify-hook

run one case and confirm the trajectory recorder fires once per iteration

rb version

print the version

rb workspace

the paid workspace: status, checkout, push

• rb workspace status
• rb workspace checkout
• rb workspace push

Exit codes

Errors

Every error carries a code, a cause and a fix. The same table is in your terminal: rb docs --errors.

E_IMPORT_INVALID

The results file is not a valid comparison.Fix the listed problems, or start from the minimal example: `rb schema example`.

E_IMPORT_NOT_JSON

The file is not valid JSON.Check for a missing comma or bracket, or start from `rb schema example`.

E_IMPORT_TOO_LARGE

The results file is over 2 MB (the limit is 500 cases).Split the case set or drop per-frame series.

E_IMPORT_CONFIG

CSV import needs the project, model names, dataset, metric and unit.Pass --project, --baseline-name, --candidate-name, --dataset, --metric and --unit.

E_CSV_INVALID

The CSV could not be converted.Required columns: case_id, baseline_error, candidate_error. Optional: name, tags, baseline_trajectory, candidate_trajectory, baseline_frames, candidate_frames (semicolon-separated numbers).

E_WORKSPACE_NO_TOKEN

No workspace token. The workspace is the paid feature; the tool itself needs no account.Set RB_WORKSPACE_TOKEN to a token from your workspace settings. `rb plans` needs no token and shows what a workspace costs.

E_WORKSPACE_AUTH

The workspace rejected this token.It may have been revoked, or it may belong to a different deployment. Create a new one in your workspace settings and set RB_WORKSPACE_TOKEN.

E_WORKSPACE_PAYMENT_REQUIRED

This workspace has no active subscription.Comparing, saving checks and failing CI stay free and local. Collecting results into a shared workspace is the paid part: `rb workspace checkout` returns a link for someone to approve.

E_WORKSPACE_NOT_SELLABLE

That plan is published but not yet sellable.It is not finished, and we do not take payment for unfinished work. `rb plans` shows which plans can be bought today.

E_WORKSPACE_UNREACHABLE

Could not reach the workspace.Check network access to the host, or set RB_WORKSPACE_URL if you run your own deployment. Nothing local was affected; the run is still on disk.

E_WORKSPACE_REJECTED

The workspace refused the request.The message says why. If it mentions the bundle's shape, push the bundle.json that `rb run` or `rb import` wrote, unmodified.

E_FILE_NOT_FOUND

A file given on the command line does not exist.Check the path; it is relative to the current directory.

E_RUN_NOT_FOUND

No such run.`rb runs` lists runs (add --runs-dir if the runs live elsewhere); ids look like 20260925-1412-raft-small. A path to bundle.json or a v1 comparison JSON also works.

E_CASE_NOT_FOUND

That case id is not in this run.`rb findings <run> --filter all` lists every case id.

E_CHECKS_INVALID

The checks file is not valid.Expected version 1 or 2; see `rb schema checks`.

E_CHECKS_PROJECT_MISMATCH

The checks file belongs to another project.Use the same project name when importing, or point --checks at that project's file.

E_LIMITS_INVALID

A limit is out of range.--max-regression ≥ 0, 0 ≤ --max-late-share ≤ 1, 0 ≤ --max-reversals ≤ 64.

E_NOT_AVAILABLE

This command is not part of this version of rabbit-brain.`rb rerun`, `rb open` and `rb mcp` are planned. Use `rb run` (or `rb import` for results you already have) and `rb findings`.

E_CONFIG_MISSING

No rb.toml in the current directory.Run `rb init --project <name> --adapter raft --model-code ./raft --dataset <path>` in the project root, or `rb init --demo` for a synthetic project.

E_CONFIG_INVALID

rb.toml could not be read.Fix the field named in the message; `rb init --force` rewrites a fresh file.

E_ADAPTER_IMPORT

The adapter could not be imported.Run `rb doctor`; install the model's own requirements (torch, opencv, scipy for RAFT) in this environment; check [adapter] id or module in rb.toml.

E_MODEL_CODE_MISSING

The model code directory is missing or not a RAFT checkout.Point [adapter] model_code at the repository checkout (it must contain core/raft.py).

E_CHECKPOINT_NOT_FOUND

A checkpoint is missing or does not match the configured architecture.Check the path; for raft-small weights set [adapter] small = true. Do not download weights without asking the human.

E_DATASET_EMPTY

No cases were found.Check [dataset] path and kind in rb.toml (KITTI: image_2/*_10.png and *_11.png; flow_occ/ optional) or the --cases file.

E_HOOK_NOT_REACHABLE

The update loop is not instrumented; no trajectory was recorded.For RAFT-family models the built-in adapter hooks model.update_block automatically; for a custom adapter call rec.step(delta) once per iteration inside infer(). Or run with --no-trajectories (stability is then 'not assessed').

E_DOCTOR

rb doctor found a problem with the environment or the project.Read data.checks: every failed check has a detail and a fix.

E_ADAPTER_DISAGREES

The adapter's per-case error disagrees with the model repository's own evaluation on the same cases, so its results are not evidence about the model.Run `rb verify-adapter --checkpoint <path>` and compare the two columns: the adapter's loading, preprocessing (input range, padding, colour order), forward call or metric formula differs from the reference path. Fix the adapter; `rb run --skip-reference` runs anyway and says so in the receipt.

E_HOOK_LENGTH

The recorder fired a different number of times than the configured iterations.Check that the hook fires exactly once per refinement iteration, and that [adapter] iterations matches what infer() runs.

E_DEVICE

The requested device is not available.Set [adapter] device = "cpu" (slow) or pass --device cpu, or run on a machine with CUDA.

E_INFERENCE_FAILED

Inference failed on every case.Run `rb verify-hook --checkpoint <path>` to see the first error; check checkpoint/architecture and the dataset.

E_WRITE_FAILED

A file could not be written.Check permissions on the current directory or pass --runs-dir.

E_EVIDENCE_DEPS

Evidence rendering needs numpy and pillow.pip install numpy pillow (both come with pip install 'rabbit-brain[raft]').

E_INTERNAL

Unexpected failure.Re-run with --json and report the output at https://github.com/rabbit-brain/rb/issues.

For coding agents

rb docs prints the full agent manual: the workflow, the file format, the JSON output of every command, exit codes and every error with its fix. Point your agent at it, then say something like “Review candidate checkpoint B against A on this case set with Rabbit Brain.”

rb docs             # the manual
rb docs --errors    # just the error table
rb <command> --json # machine-readable output, on every command

The same manual is AGENTS.md in the repository, so an agent with repo access can read it without installing anything.