Crucible · Advanced · Tutorial 3 of 3
Thirty runs. A third of the frame rate gone.
No regression reported.
Batch mode runs every scenario unattended, records what it measured, and tells you whether you got slower. It is genuinely good at that — for one kind of slower. This page is about which kind, why the other kind is invisible, and what to read instead.
What a batch run is
Every scenario in turn: build the world, let it settle for 30 frames, sample the next 30, average them, compare against history, write it all down.
The frames are real — it waits on the scene tree's own frame signal rather than simulating a clock, so physics settles and draw calls are the draw calls you would actually make.
The one design decision
There is no reference machine and no target frame rate. The baseline is the median of your own previous runs of that scenario, on this machine.
Everything else on this page — the strengths and the blind spot alike — falls out of that single choice.
The detector sees cliffs. It cannot see slopes.
Measured: thirty consecutive runs each 9% below the current bar produced zero regression reports while the frame rate went from 60 to 37.7 — a 37% cumulative loss. Each run was under the threshold, and each run then became part of the baseline the next one was judged against.
Sampling
What one run measures
Each scenario gets a warmup window and a sample window, both 30 frames by default and both adjustable.
| Recorded | How |
|---|---|
| FPS, frame ms, draw calls, triangles | Averaged over the sample window. |
| Video memory, static memory, object count | Taken from the last frame only, not averaged. |
The split is sensible — averaging a memory figure over 30 frames tells you less than reading it once at the end — but it is worth knowing which numbers in a report are means and which are single observations, because they behave very differently when something is spiking.
The warmup exists because the first frames after a world is built are not representative: physics is still settling, meshes are still being uploaded. If a scenario of yours is unusually heavy to construct, raise the warmup before concluding it is slow.
The baseline
The median, and why
Not the mean, not the best, not the last: the median of every prior recorded run of that scenario.
| History | Baseline |
|---|---|
| nothing yet | null — and null is never a regression |
| 100, 50, 60 | 60 (the mean would be 70) |
| 40, 60, 80, 100 | 70 — midpoint of the middle two |
| 60, 60, 60, 60, 3 | 60 — the disaster barely registers |
That last row is the whole argument for a median. You compile something in another window, or leave the editor open, or a backup kicks off — and one run comes back at 3 FPS. A mean would drag the bar down and quietly excuse a real regression next week. The median absorbs it.
The current run is not part of its own baseline: the comparison happens first, the append happens after. And the baseline is per scenario id, so scenarios never contaminate each other — with one exception, in lesson 05.
The verdict
The threshold
A run regresses when its FPS falls more than 10% below the baseline.
| Baseline 100, this run | Verdict |
|---|---|
| 89 | Regression |
| 90 — exactly 10% down | Not a regression. The comparison is strict. |
| 91 | Not a regression |
| 200 | Not a regression, and not celebrated either |
| no baseline | Not a regression — the run becomes the bar |
The 10% is a constructor argument, not a constant. A detector built at 2% flags a 3% drop, so if you are tuning something specific you can hold it to a much tighter standard than the default.
⚠ One detail that matters if you consume the JSON. A run with no baseline reports its
percent change as 0.0 — the same value a genuinely unchanged run
reports. The two are only distinguishable by baselineFps being null. A dashboard
that plots percent change without checking that will draw a flat, reassuring line for every
scenario it has never seen before.
The blind spot
Cliffs, not slopes
Put the two previous lessons together. Each run is judged against the median of everything before it — and then joins that median.
| Shape of the loss | What batch mode does |
|---|---|
| One build drops you 20% against a flat history | Caught immediately. |
| Thirty runs, each 9% under the current bar | Zero regressions. 60 → 37.7 FPS. |
Every single one of those thirty runs was under the threshold, honestly, and every one of them then moved the bar down for the next. The bar walks down with you.
The decay is slower than "9% per run" would suggest — a 37% total loss rather than the 94% that compounding would give — precisely because the median is sticky. All those early fast runs are still in the file, holding the bar up while reality falls away from it. The robustness that protects you from one bad sample is the same property that hides a trend, and it cannot be tuned to do one without the other.
So a long refactor, a slowly growing asset budget, a scenario that gains one more agent each week — none of these are what this tool is looking for. It is a smoke alarm, not a thermometer.
user://history/<id>.jsonl, one JSON object per line, oldest first. Plot the
fps field against utc and a slope is obvious in one glance — the
data was never missing, only the alarm. If you want a hard floor instead, construct the detector
with a tighter threshold and compare against a figure you chose rather than one you drifted to.Storage
Where history lives
One append-only JSONL file per scenario under user://history, one
line per run, never trimmed.
It sits outside the mod, which is the right call and worth distinguishing from the
user:// habit other tools have. A measurement is machine-specific: your frame rate
is not your teammate's, and a shared baseline would be meaningless. This is not content that
failed to reach its destination — it is a record that belongs where it is.
Two consequences. A fresh clone starts blind, with no baselines, so the first run of everything is a free pass. And the file grows forever, which is what makes the median progressively harder to move.
Filenames are sanitised: every character that is not a letter, digit, underscore or hyphen becomes an underscore.
So my.scene and my_scene write to the same file and share
one baseline. Verified: recording 10 for one and 90 for the other gives both a baseline
of 50. Dots, spaces and slashes in scenario ids all collapse this way. Keep ids in plain
snake_case and it cannot happen.
One good behaviour to know about: a corrupt line — a half-written entry from a run that was interrupted — is skipped rather than fatal. You lose that line and nothing else, and the baseline is computed from the rest.
Output
The two reports
Every report is written twice, to user://output, timestamped.
| File | For |
|---|---|
| Styled HTML | Reading. Open it, skim it, send someone a screenshot. |
| JSON twin | Everything else — CI, dashboards, diffing two runs. |
The pairing is the useful part: you never have to choose between a report you can read and a report you can process, and the two always describe the same run. Validation reports follow the same pattern, so a full session leaves you a readable record and a machine-readable one of both what was wrong and what was slow.
Names are timestamped to the second — batch_20260831_143002.json — so runs
accumulate rather than replacing each other, and nothing prunes the directory for you.
⚠ To the second, though, not finer. Two reports of the same thing completing inside the same wall-clock second land on the same filename and the second one wins. A batch run is long enough that this will not happen to you; a validation report is not, and its name is just the scene slug plus that stamp — so reload a scene twice quickly and the first report is gone. If you are keeping one as evidence, copy it out rather than trusting the name to be unique.
Ladder complete
What you now know
Check yourself
- A batch run gives each scenario 30 warmup frames and 30 sampled ones, on the real frame signal. FPS and draw counts are averaged; memory and object count are last-frame only.
- The baseline is the median of prior runs of that scenario on this machine — computed before the new result is appended.
- A median means one catastrophic sample barely moves the bar. That is the design's main strength.
- A regression is more than 10% below baseline, strictly — exactly 10% does not count. The threshold is injectable.
- No baseline is never a regression, and reports
0.0%— indistinguishable from "unchanged" unless you checkbaselineFpsfor null. - The detector catches cliffs, not slopes. Thirty runs at 9% each cost 37% of the frame rate and raised no flag.
- History is append-only JSONL under
user://history, per machine, never trimmed — and ids sanitise into each other, somy.sceneandmy_sceneshare a baseline. - Reports come as an HTML and JSON pair, timestamped to the second, so a fast repeat can overwrite.
That is the Crucible ladder. You can read a validation report for what it actually claims, build a sandbox that does what you meant rather than what it silently defaulted to, and run the whole thing unattended while knowing which kinds of slower it will tell you about and which kind you have to go and look for.