Skip to content

Phase 2: the baseline measurement window

Parent plan: PLAN-ci-cloud-sizing.md.

Planning effort: high, as the master plan specifies. Phase 1 built the instrument; this phase decides what its output means, and the whole plan's remaining phases are argued from the numbers this one publishes. The judgement is not in the arithmetic -- it is in choosing which statistic the sizing decision is allowed to rest on, and in being honest about the parts of the measurement that are still blind.

Decision numbering continues the plan-wide sequence: phase 0 used D1-D8 and phase 1 used D9-D15, so this phase begins at D16.

Context

The master plan was written from three hand-collected data points and says so. Phase 1 replaced them with an instrument: every functional cluster job now samples /admin/resources and the node roster every 15 seconds, takes a filtered Loki census of the scheduler's per-candidate stage events, and prints a summary into the job log, with the raw series and census landing in the 90-day bundle.

Phase 2's job, in the master plan's words, is to "leave phase 1 running for an agreed number of merge runs, then publish the distribution: what peak utilisation actually is per job, on both topologies, and how it correlates with the failures". That distribution is what turns the candidate shapes in the master plan into a decision, and it is what phases 3, 4 and 5 all draw on.

The survey below found that the waiting is already done.

Scope

In scope:

  • A harvest tool in this repository which turns the banked bundles from many merge runs into one machine-readable dataset.
  • A machine-readable summary record emitted by phase 1's report tool, so the harvest and phase 5's guardrail read the same contract rather than each parsing prose.
  • Publishing the capacity read's degraded flag through /admin/resources, because without it roughly 8% of samples are discarded for a reason nobody can name.
  • Correcting the refusal census filter in shakenfist/actions so the capacity guard events are actually collected.
  • The baseline itself: the distribution, written into the master plan, with the band bounds either defended or replaced (D3), the D7 ledger reconciliation closed, and open question 5 answered against data rather than against a single run.
  • Issues filed for anything the measurement exposes that sizing cannot fix.

Out of scope, deliberately:

  • Any topology change. That is phase 4, and phase 3 gates it.
  • Any change to what the scheduler admits. If the ledger-unreadable window turns out to be a real defect, this phase files it and phase 4 does not depend on it being fixed.
  • Turning the band into a gate. Phase 5 owns that; this phase only supplies the numbers it will gate on.
  • Documenting the sizing model in docs/developer_guide/ci.md. That is phase 6. The harvest tool gets a paragraph, because it is a tool a reader will otherwise not know exists.
  • The under-cloud probe, per-suite concurrency, and the other master-plan Future work entries.

What the survey found

Eight findings. Four contradict claims this phase inherited and are corrected at source in the planning commit -- in the master plan's Situation and phase 2 sections -- so the next reader does not trip over them. Re-check those rather than redoing them.

Findings 1 to 7 come from the planning survey. Finding 8 was added later, during implementation, when verifying step 2c against live merge runs turned up something the planning survey could not have seen without the harvest existing.

1. The waiting is already done: 66 merge runs are banked. The master plan's phase 2 section reads as though a window must be opened and waited out. It need not be. The census-filter fix (shakenfist/actions PR #45) merged 2026-08-30, and gh api "repos/shakenfist/shakenfist/actions/workflows/functional-tests.yml/runs?event=merge_group" returns 66 runs created since then, each carrying five cluster bundles at roughly 5 MB apiece. Artifact retention is 90 days (smoke-cluster.yml:584), so the whole window is downloadable today and none of it is downloadable in December. The measurement this phase exists to publish is a harvest, not a wait. D16.

2. The instrument works, on both topologies, and its first two readings already disagree with the plan's headline statistic. Merge run 33944911413 (2026-09-05) printed a complete summary for both shapes:

samples cluster p90 frac cluster peak frac ledger
slim-primary 131 0.407 0.481 27.0
slim-tier 133 0.667 0.833 12.0

Read cluster-wide, slim-primary looks comfortable. Read per node, it is not: node 12958f0f sat at p90 1.000 and peak 1.000 of its own six-vCPU ledger for the whole run, and 0c1c16b9 (the three-vCPU network node) peaked at 1.000, while two other nodes peaked at 0.500. The same run recorded 12 candidate drops at sufficient_idle_cpu, reason "would exceed hard max CPUs" -- real refusals, in a job whose cluster-wide utilisation was 41%. On slim-tier all three nodes peaked at 1.000.

A cluster-wide fraction averages a full node against an empty one and reports the mean as headroom. That is the statistic D3's band is currently written against. This is the evidential basis for D21, and it is the most decision-relevant thing the instrument has said so far. Step 2d defended it across the window -- the per-node maximum exceeds the cluster-wide figure by a median of 2.25x -- with one correction: the statistic saturates, pinning in 100% of tier job-runs, so it discriminates nothing at the top and phase 5 must read it as a goal rather than an alarm.

3. Roughly 8% of samples are discarded, and nothing can say why. Both runs report a LEDGER UNREADABLE block: 10 of 131 samples on slim-primary and 12 of 145 on slim-tier had cpu_committed_row_present false for every node at once. Those samples are excluded from every committed-CPU figure.

The cause is not knowable from the data. _capacity_by_node() (shakenfist/scheduler.py:234) returns (capacity, degraded), and the degraded flag distinguishes a failed read from an unpopulated table -- GetSchedulerNodeCapacity (shakenfist/daemons/database/main.py:2760) forwards it precisely so that the two are not confused. But summarize_resources() throws it away at scheduler.py:1005 (capacity, _ =), with a comment explaining that the summary has no instance to record an event against. That reasoning is sound for an event; it is not a reason to omit the flag from the response body. Phase 1's D12 added cpu_limit to the same block for the same class of reason.

This matters beyond the measurement. During such a window the CPU pre-filter charges every node zero and compares against no limit, so the cheap pruning in front of the admission guard is blind -- which is exactly the window in which a burst of concurrent creates would be admitted and then refused by the guard. Whether that is happening is not assertable today. D19 publishes the flag; step 2f files the issue if the flag says the reads are failing.

4. The capacity guard census is structurally empty, and the report says so itself. tools/ci_headroom_collect.sh in shakenfist/actions queries {job="shakenfist"} |~ "schedule (at stage|has no candidates at stage)". That regex matches the scheduler's stage events and nothing else. The two guard messages -- instance placement denied (shakenfist/instance.py:1303) and placement admitted over namespace capacity claim (:1209, :1218) -- do not match it, so every census in the banked window carries zero guard events. The report prints a Capacity guard census section which says outright that this is a fact about the query before it is a fact about the cluster, and names both missing strings. Phase 1's own output caught this. D20 fixes the filter; the retrospective window keeps the hole and reports it as unknown rather than zero.

5. The tier's ledger is 12, and the "10 by observation" discrepancy no longer exists. The master plan records the slim-tier ledger as "12 by derivation, 10 by observation", citing #3907, and phase 0's D7 made reconciling the two this phase's job. Both halves now say 12. The live series reports a cluster ledger of exactly 12.0 with per-node limits of 3, 6 and 3, and 399 node-samples carrying a real capacity row against zero fallbacks to cpu_hard_max. #3907 itself closed COMPLETED on 2026-08-27, and its final recurrence comment quotes the cluster singleton refusing a claim with cpus (limit 12, used 9, requested 4) -- the singleton says 12 too. D7 is closed by evidence already in hand. Corrected at source.

Two caveats the harvest must still honour. The singleton's total is not published by /admin/resources (summarize_resources() builds total from per-node arithmetic only), so the harvest confirms the sum of the rows and cites #3907 for the singleton rather than measuring it. And "zero fallbacks" is over two runs; the harvest reports the fallback count across the window, because a single node without a row would change what the ledger column means.

6. Memory is nowhere near binding, which puts D5 in question. Phase 0's D5 decided memory is "a real second dimension" on the strength of one post-#3813 run refusing at sufficient_idle_memory. In both survey runs the memory stage recorded 144 events and zero drops, and committed memory ran at a p90 of 0.136 of ledger on slim-primary and 0.199 on slim-tier -- against 0.407 and 0.667 for CPU. Two runs are not a distribution, which is exactly why this is listed as a finding to test rather than as a correction. But the master plan's phase 2 section says to "expect it to answer open question 5", and the expected answer has flipped: the harvest is now testing whether D5 should be narrowed, not whether it holds.

7. The report has no machine-readable output, and the series carries no topology label. tools/ci_headroom_report.py takes --series, --census, --label and --census-limit (:1630-1656) and prints prose. A harvest over 300-odd bundles cannot parse that, and phase 5's guardrail should not either. D18 adds the record. Separately, the topology is passed to the report at run time (--label "${topology} ${stestr_config}") and never written into the series, so a bundle on disk does not say which shape produced it; the artifact name does (bundle-shakenfist-full-debian-12-slim-tier), but not for guests or ansible-modules, which are slim-primary per functional-tests.yml:462,511. The harvest uses an explicit table that fails loudly on an unknown bundle, and D20's actions change writes a label file so future harvests do not need one.

8. A node's ledger can read as 200% committed, and that alone would produce the 507 family. Found while verifying step 2c against two live merge runs, so it is an observation from the instrument rather than from the plan. In run 33948911843's Debian 12 tier job -- which failed -- the raw sample for one node reads cpu_measured: 0, cpu_committed: 6, cpu_limit: 3, cpu_committed_row_present: true. Measured is zero, so this is not a measurement artefact and not overcommit of real CPU: the materialised capacity row itself records the node as holding twice the vCPU its own limit allows, with no domains running. The limit did not move during the run -- the harvest records the ledger as the range it moved over, and that range is 3.0 to 3.0.

The consequence is mechanical. _has_sufficient_cpu() admits on max(measured, committed) + requested <= limit, so a node in this state refuses every subsequent create regardless of how idle it is, and the cluster loses a third of its hypervisors without anything reporting an error. That is the #3772 507 signature exactly, arrived at from the capacity counters rather than from real exhaustion -- and it is a defect that growing the cloud would mask rather than fix, which is precisely the outcome phase 3 exists to prevent.

Resolved by step 2d, and more narrowly than this finding expected: it occurs in 18 of 85,563 node-samples, inside 2 of 204 job-runs, and one of those two job-runs passed. At one percent it cannot explain the tier's failure rate and does not. It remains a genuine invariant violation, filed by step 2f as #4087, and it remains a defect a bigger cloud would mask -- but the framing below, written before the window was harvested, overstated it. See What the baseline measured.

This phase does not chase it: steps 2d and 2f own it, and it is recorded here so that it cannot be lost. What 2d must establish is how often the window contains it, and whether it correlates with the failing jobs; what 2f must establish before filing is how a row reaches that state at all, given that admit_instance_placement()'s guarded UPDATE is supposed to make used + requested <= limit an invariant. The likely candidates are a reconciler faithfully recomputing used_cpus from instances that are genuinely placed there, having arrived when the limit was higher or by a path that did not go through the guard.

Step 2f established it, and it is the second of those. The placements arrived by a path that did not go through the guard because there was no guard yet: scheduler_node_capacity has no rows until the reconciler's first pass, all 18 node-samples are the first samples after that pass, and every admission before it takes P7's fail-open branch. The full account is in the master plan's A node can record twice its own ledger, and sizing would hide it.

Decision items

D16 -- Harvest the banked window; do not open a new one

Decision: the baseline is computed retrospectively from the merge runs already banked since 2026-08-30, not by agreeing a number of future runs and waiting for them.

Reasoning: the master plan wrote phase 2 as a waiting exercise because, when it was written, no instrument existed. One has existed for a week, and survey finding 1 counts 66 merge runs of output sitting in artifact storage. Waiting would add weeks and no information. It would also lose information: retention is 90 days, so the early part of the window expires while we wait for the late part. The one thing a prospective window buys -- data recorded after D19 and D20 land, and therefore free of findings 3 and 4's blind spots -- is bought much more cheaply by step 2g's short confirmation window, which needs a handful of runs rather than a statistically useful number.

What a reviewer might say instead: that a retrospective window measures a moving target, since the tree changed under it across 66 runs. That is true and is the reason the harvest records the head SHA per run and the report is run per job rather than pooled -- a step change part-way through the window is visible as one, and git log explains it. A prospective window has the same problem with a smaller n.

D17 -- Every instrumented cluster job, not only the two named topologies

Decision: the harvest covers every job that actually carries the phase 1 probe, reports per job as well as per topology, and treats the set as exactly these four per merge run:

Job Topology Bundle artifact
Debian 12 cluster slim-primary bundle-shakenfist-full-debian-12-slim-primary
Ubuntu 24.04 cluster slim-primary bundle-shakenfist-full-ubuntu-2404-slim-primary
Guests slim-primary bundle-shakenfist-full-guests
Debian 12 tier slim-tier bundle-shakenfist-full-debian-12-slim-tier

The job column is the readable name the matrix gives and the name step 2d groups by. It is not what the jobs API returns: the reusable workflow contributes its own name, so the API reports Debian 12 cluster (collection) / Smoke tests (collection). The harvest records both, matching on the prefix, and records a null job conclusion rather than falling back to the run's when the match fails. Found while implementing step 2c.

Corrected after this plan was committed. This decision originally named six jobs, adding Ansible modules and Node lifecycle. Neither is instrumented, and both were checked empirically against merge run 33944911413 rather than reasoned about: their bundles contain no traces/ directory at all. The two causes are different. Ansible modules runs through the reusable smoke-cluster workflow but with test_kind: ansible-modules (functional-tests.yml:514), and every probe step is gated if: inputs.test_kind == 'functional'. Node lifecycle never reaches that workflow: it calls the build-smoke-cluster composite action directly (functional-tests.yml:554-557), so the probe steps, which live in the workflow rather than the action, are not in its job at all.

Consequence for the baseline. The master plan's failure table spans six jobs, from 92% for node lifecycle down to 19% for the tier, and the harvest can speak to only four of them. Step 2d must say so rather than presenting a four-job dataset against a six-job table. In particular node lifecycle, the best-performing job in that table, is unmeasured, so no claim of the form "utilisation explains the pass-rate spread" can be made across the whole table.

Reasoning for the decision itself, unchanged: three of the four instrumented jobs run the same topology. If slim-primary's jobs differ from each other in peak demand, the difference is the suite, not the shape, and phase 4 must not respond to it by resizing the cloud. Pooling by topology would hide that. It costs nothing, since the bundles are downloaded per run anyway.

Not in scope to fix: instrumenting the other two. Node lifecycle would need the probe steps moved into or duplicated beside the composite action, which is a change to how every caller deploys; Ansible modules would need the gate widened. Both are worth doing and neither is this phase's business -- they are recorded in the master plan's Future work by step 2f.

D18 -- One summary record, two consumers

Decision: tools/ci_headroom_report.py gains a summary_record() function returning a plain dict, and a --json PATH option which writes it. The prose summary is rendered from the same record. The harvest calls the function; phase 5's guardrail will read the file.

Reasoning: the alternative is parsing the printed tables, which makes every future change to the report's wording a silent break of the harvest. Deriving both outputs from one record means the number in the job log and the number in the dataset cannot disagree -- which is a real risk here, because the whole plan turns on people trusting these figures. The function form rather than only a CLI flag is so the harvest, which runs locally in this repository, need not shell out 300 times.

Constraint carried from phase 1: the tool still has to run on the CI runner under stock python3 with no third-party import. The record is built from standard-library types only.

D19 -- Publish the capacity read's degraded flag

Decision: summarize_resources() publishes capacity_degraded in the response's total block, sourced from the second element of _capacity_by_node() which it currently discards. The probe records it like everything else, and the report classifies a ledger-unreadable sample as a failed read or an unpopulated table accordingly.

Reasoning: survey finding 3. Eight percent of samples are being dropped for a reason the response makes unknowable, and the flag that would answer it is already computed one line above the point where it is thrown away. This is the same shape of change as phase 1's D12, for the same reason: the field exists to let a reader see which of two indistinguishable conditions actually happened.

What this does not do: it does not act on the flag, log anything, or change admission. The discard comment at scheduler.py:1000-1004 is correct that this summary has no instance to record an event against, and that stays true. Only the response body changes.

D20 -- Fix the census filter, and label the series

Decision: the LogQL filter in tools/ci_headroom_collect.sh (shakenfist/actions) is widened to match the two capacity guard messages alongside the stage events, and the collect script writes the topology label into /srv/ci/traces/headroom-label so it lands in the bundle. The retrospective window is reported with guard refusals as unknown, never as zero.

Reasoning: survey finding 4. The report already tells the reader this section is meaningless and names the exact strings missing from the query; leaving it unfixed for another phase means phase 3, which is about what happens at the capacity boundary, plans against a census that cannot see the boundary being enforced. The label is a two-line addition to the same script and removes the one place where the harvest has to guess.

Cost: it lands in shakenfist/actions, which cannot exercise a composite-action change in its own pull request, and only the operator can push. Phase 1 D15's discipline applies unchanged -- continue-on-error, script always exits 0.

D21 -- The band is per node as well as cluster-wide

Decision: D3's headroom band gains a per-node component. The band is evaluated against both the cluster-wide committed fraction and the maximum per-node committed fraction, and a run is outside the band if either is. The numeric bounds for both come from the harvest; the provisional 0.35/0.70 are treated as bounds on the cluster-wide figure only until the distribution replaces them.

Reasoning: survey finding 2. On slim-primary, a cluster-wide p90 of 0.407 -- comfortably inside the provisional band -- coexisted with one node pinned at 1.000 for the entire run and twelve real sufficient_idle_cpu refusals. The cluster-wide statistic is an average over nodes, and the scheduler does not admit against an average: it admits against one node's ledger at a time. A band written only cluster-wide would have declared that run healthy, which is the precise failure mode this plan exists to stop making.

This is the decision most likely to be argued with, on the grounds that phase 2 is meant to supply the distribution and let phase 5 decide the band, not amend the band's form. The answer is that D3 fixed the band's form before any distribution existed, the first two samples of that distribution show the form is wrong, and a phase which published a distribution against a statistic it already knew to be misleading would be doing the reader a disservice. The numbers still come from the harvest; only the shape of the question changes. If the harvest shows the pinned node is an artefact of those two runs, step 2d records that and D21 is narrowed rather than defended.

Secondary reasoning: it also gives phase 4 a target it can act on. "The cluster is 41% used but one node is full" argues for widening nodes, which is D1's decision; "the cluster is 41% used" argues for shrinking it, which is the opposite.

D22 -- The deliverable is prose in the master plan, plus a dataset

Decision: the baseline lands as a rewritten Situation section in the master plan -- replacing the hand-collected figures with measured ones -- plus the harvest dataset committed under docs/plans/data/ci-cloud-sizing-baseline/. The phase does not propose a topology.

Reasoning: the master plan says phase 2's output "is what turns the candidate shapes above into a decision", and phase 4 makes that decision after phase 3 gates it. Writing a proposed shape here would front-run both. Committing the dataset rather than only the conclusion is what lets phase 4, phase 5 and a future re-measure check the arithmetic instead of trusting it -- and the bundles it came from expire in 90 days.

Size constraint: the dataset is the per-job summary records, not the raw series. The raw series stays in the bundles and is not committed.

Corrected after this plan was committed. This paragraph originally guessed "a few hundred bytes each... low hundreds of kilobytes". A real record, measured by running the step 2b tool over a real bundle from merge run 33944911413, is 3,675 bytes compact and 5,381 indented -- an order of magnitude out. With D17's corrected job count that is 66 runs x 4 jobs, and the window grows with every merge run before step 2d executes. Step 2c then measured eight harvested records -- which wrap the report record in framing fields -- at 5.0 KB compact on average, giving roughly 1.3 MB for the window. That is inside the budget below but with much less slack than either earlier estimate suggested.

So: the harvest output is serialised compact (separators=(',', ':')), one record per line, and the budget is 2 MB rather than the one megabyte the definition of done originally demanded. If it exceeds that, prune before relaxing it again -- the bulk is absences.classifications[].nodes and the guard block, and the node lists are the part a re-analysis is least likely to need. A single job's --json output keeps indent=2, because that one is read by a human.

Step plan

Step Effort Model Isolation Brief for sub-agent
2a medium sonnet none Publish the capacity read's degraded flag from Scheduler.summarize_resources() in shakenfist/scheduler.py (D19). The value is already computed and thrown away: line 1005 reads capacity, _ = self._capacity_by_node(). Bind the second element and set resources['total']['capacity_degraded'] from it, in the total dict initialised at :1008. Do not log, raise an event, or change admission -- the comment at :1000-1004 explaining why a degraded read is not acted on here stays true and should be extended, not deleted, to say that the flag is now published for the reader instead. _capacity_by_node() at :234 documents the two-value contract; the scheduling path at :661 shows the other caller. Add a unit test beside test_summarize_resources_publishes_the_counters (shakenfist/tests/test_scheduler.py:805) pinning both the degraded and non-degraded cases; look at how test_summarize_resources_reports_a_node_with_no_row (:821) seeds capacity state via mock_mariadb to find the hook for forcing a degraded read. The swagger example in shakenfist/external_api/admin.py is elided (admin_resources_get_example at :98 is literally {...}), so it needs no change. Single quotes, 120 columns.
2b high opus none Add a machine-readable summary record to tools/ci_headroom_report.py (D18). Factor the computation the printer already does into a summary_record(series, census, label, census_limit) returning a plain dict, and render the existing prose from that record so the two cannot disagree; add a --json PATH argument writing it as one JSON object. Do not change a single line of the printed output -- phase 1's tests pin behaviour and a reader comparing an old job log to a new one must see no difference. The tool is 1677 lines with 50 tests in shakenfist/tests/test_ci_headroom_report.py; run them. The record must carry, at minimum: the label, sample counts (usable, failed, unparseable), the window start/end/duration, cluster-wide p90 and peak for committed vCPU and memory both absolute and as a fraction, the same per node keyed by node uuid including that node's ledger, the ledger-provenance counts (rows present, fallbacks, fallbacks inside unreadable samples, no ledger at all), the ledger-unreadable sample count, the per-stage census tally with aborts and drops and the drop reasons, the guard census state (present versus not collected -- never zero when not collected, per D20), the D3 band verdict, and the new per-node maximum fraction D21 introduces. Two hard constraints from phase 1 carry over unchanged: standard library only, because this runs on the CI runner under stock python3; and the tool exits zero always (D15), so a --json path that cannot be written prints a warning and does not raise. Add tests for the record: that every number in the record matches the number in the printed prose for the same input, and that an absent census yields the not-collected state rather than zeros.
2c high opus none Write tools/ci_headroom_harvest.py in this repository (D16, D17). It enumerates merge_group runs of functional-tests.yml, downloads each run's functional cluster bundles, extracts headroom.jsonl and headroom-census.json, calls 2b's summary_record() on each, and writes one JSON object per job per run to an output file. The bundle is a nested zip and this was verified against a real artifact, not assumed: the artifact download is a zip whose single entry is bundle.zip, and the files you want are inside that at bundle/traces/headroom.jsonl and bundle/traces/headroom-census.json. A sample record is {'sampled_at': ..., 'resources': {'per_node': ..., 'total': ...}, 'nodes': [...]}, and the census is a Loki query_range response with its streams under data.result. Use gh api "repos/shakenfist/shakenfist/actions/workflows/functional-tests.yml/runs?event=merge_group&per_page=100" for the run list and .../runs/<id>/artifacts for the bundles, both via subprocess on the gh CLI rather than a HTTP library -- this is a local developer tool, so unlike the report it may use third-party imports, but gh handles the auth. Bundles are zips at roughly 5 MB; download to a cache directory keyed by artifact id and skip anything already cached, because this will be run more than once. The topology is not in the series (survey finding 7): map it from the artifact name with the explicit four-row table in D17, sourced from the matrix at .github/workflows/functional-tests.yml:436-480, and fail loudly on a bundle name the table does not cover rather than guessing or skipping silently. Only four bundles per run carry the probe at all -- bundle-shakenfist-full-ansible-modules and bundle-functional-node-lifecycle-collection have no traces/ directory, for the two different reasons D17 gives -- so the harvest should skip those two by name, deliberately and with a comment, rather than treating them as missing data. Record per record: run id, head SHA, run creation time, job name, topology, the run's conclusion and that job's conclusion, and the summary record. A bundle with no headroom.jsonl (a run predating phase 1, or one whose probe never started) is recorded as such and not dropped. Add unit tests over a fixture bundle zip; follow shakenfist/tests/test_ci_headroom_report.py for how it loads a tools/ script by path with importlib.util.spec_from_file_location. Document the tool in one paragraph in docs/developer_guide/ci.md, beside phase 1's headroom section -- the sizing model itself is phase 6, so do not document topologies here.
2d xhigh opus worktree Run the harvest over every merge run since 2026-08-30 and write the baseline (D22). This is the phase. Commit the dataset to docs/plans/data/ci-cloud-sizing-baseline/ (summary records only, not raw series -- D22) with a README naming the window, the tool and the command. Then rewrite the master plan's Situation section so every figure in it is measured rather than hand-collected, keeping the section's structure and its cross-references intact. It must answer, with numbers and with n stated for each: what peak and p90 committed vCPU actually are per job and per topology; how the per-node maximum fraction relates to the cluster-wide one (D21 -- if the pinned-node pattern in survey finding 2 does not hold across the window, say so plainly and narrow D21 rather than defending it); whether committed-CPU utilisation correlates with the job failing, which is the master plan's central claim and is now testable -- but only across the four instrumented jobs (D17), and node lifecycle, the best performer in the master plan's failure table, is not one of them, so say what the correlation is computed over and do not present it as spanning that table; what the refusal census says per stage, with sufficient_free_disk and sufficient_idle_disk kept distinct; whether memory ever binds, which decides whether phase 0's D5 stands or narrows (survey finding 6); the ledger-provenance counts closing D7 (survey finding 5 -- the answer is already 12, so this is confirming a count of fallbacks across the window, not re-litigating); and the band bounds, either defending 0.35/0.70 for the cluster-wide figure or replacing them, plus a first proposal for the per-node bound. Be explicit about the two blind spots: guard refusals are unknown in this window, not zero (D20, survey finding 4), and the ledger-unreadable samples are excluded for a reason 2a only makes knowable prospectively (D19, survey finding 3). Do not propose a topology -- that is phase 4 and phase 3 gates it.
2e medium sonnet none Fix the refusal census in shakenfist/actions (D20). In tools/ci_headroom_collect.sh, widen the LogQL line filter so it matches the two capacity guard messages as well as the scheduler stage events: the current regex schedule (at stage\|has no candidates at stage) misses instance placement denied (shakenfist/instance.py:1303) and placement admitted over namespace capacity claim (:1209, :1218), which is why ci_headroom_report.py prints a Capacity guard census section saying it has nothing to count. The script's own header comment explains at length why the filter is a regex and why there is no \|= "Added event" line filter -- keep both explanations and extend the first. Also write the label the script is already passed into /srv/ci/traces/headroom-label on the primary, so it lands in the bundle and a later harvest need not infer the topology from the artifact name. Observe the repository's conventions: at most about five lines of inline shell per workflow step, shellcheck --severity=error clean, and nothing in this script may fail the job (it must still always exit 0, per phase 1 D15). This lands untested -- a pull request against actions cannot exercise a composite-action change -- so write it to be inert on failure, and note in the pull request that only the operator can push it.
2f medium opus none File the issues the measurement exposed, and only those. Three candidates, and each is filed only if 2d's data supports it: first, the 200%-committed capacity row of survey finding 8 -- establish how a row reaches used_cpus twice its own limit_cpus when admit_instance_placement()'s guarded UPDATE is meant to make that impossible, and file it as a scheduler defect rather than a CI one, because a node in that state refuses every create while measuring completely idle; second, the ledger-unreadable window (survey finding 3) -- if 2a's flag, once running, shows these are failed reads rather than an unpopulated table, file that the CPU pre-filter is periodically blind and say what the observed rate is; if it shows an unpopulated table, file nothing and record the finding in the plan instead. Second, anything 2d finds that sizing cannot fix, which per the master plan's phase 3 framing must be recorded rather than silently absorbed when the clouds grow. Third -- not an issue but a Future work entry in the master plan -- that two jobs are uninstrumented for two different reasons (D17): Ansible modules because every probe step is gated if: inputs.test_kind == 'functional', and Node lifecycle because it calls the build-smoke-cluster composite action directly and never reaches the workflow the probe steps live in. Note that the second is the same structural reason the master plan's Future work already gives for propagating the probe to downstream repositories, so the two entries should reference each other. Follow the repository's issue conventions and cross-reference #3772 where the signature matches. Add a Bugs fixed during this work entry to the master plan for each. Do not fix anything: this phase files and phase 4 does not depend on any of it being fixed.
2g medium opus none After 2a and 2e have merged and a handful of merge runs have used them, re-harvest a short confirmation window and add an addendum to 2d's baseline. This closes the two blind spots the retrospective window could not: what the capacity guard census actually contains, and whether the ledger-unreadable samples are failed reads or an empty table. Five to ten merge runs is enough for both -- this is a classification, not a distribution, and it must not be allowed to grow into a second measurement window. If the addendum changes a conclusion in 2d, change the conclusion rather than appending a contradiction. Then hand 2f whatever it needs to decide the first issue.
2h low haiku none Set the phase 2 row to Complete in the master plan's Execution table and the index arithmetic in docs/plans/index.md to 3 of 7, then run python3 tools/check-plan-status.py. Do this only after 2a-2g are reviewed and the operator has confirmed the addendum in 2g is based on real post-fix runs.

The survey corrections that would otherwise have been a step here were made in the planning commit, per the note under What the survey found. Note also that phase 1's row was still Not started in both status tables when this phase was planned, despite its work having merged on 2026-08-29; that is corrected in the same commit.

What the baseline measured

Recorded here so this plan stands on its own. The full write-up is the master plan's Situation section; the dataset is docs/plans/data/ci-cloud-sizing-baseline/. Step 2d ran the harvest on 2026-09-05.

The sample. 66 merge_group runs enumerated; 55 banked a functional cluster bundle (eleven did not -- three cancelled, eight skipped by Check paths); 52 banked probe output. 217 records, 204 with a usable committed-CPU series, over 2026-08-30T07:48:03Z to 2026-09-05T07:15:07Z. The effective window starts at 07:48 rather than midnight because the three earliest runs of 2026-08-30 predate the shakenfist/actions change that starts the probe -- D16's "since 2026-08-30" was a day too generous by about eight hours. The window is stationary: splitting each job in half shifts nothing (smallest p = 0.08).

Committed vCPU, cluster-wide, per job-run.

Job Topology n median p90 fraction max p90 fraction
Guests slim-primary 51 0.296 0.370
Debian 12 cluster slim-primary 52 0.333 0.444
Ubuntu 24.04 cluster slim-primary 51 0.333 0.519
Debian 12 tier slim-tier 50 0.750 1.000

slim-primary (n=154) never exceeded 0.519; slim-tier (n=50) never fell below 0.500. The three slim-primary suites differ from each other far less than from the tier, which is the answer D17 was written to make askable: the gap is the shape, not the suite.

D21 is defended, with a correction. The per-node maximum exceeds the cluster-wide figure by a median ratio of 2.25x (n=204). A node reached its own ledger in 65% of slim-primary job-runs and 100% of slim-tier ones; a node sat at its ledger for ninety percent of the run in 48% and 100% respectively. The pinned-node pattern from survey finding 2 holds across the window, disproportionately on the 3 vCPU network node (44% of its node-job-runs, against 10% for the 6 vCPU nodes). The correction is that the per-node maximum is saturated: it is at its ceiling in most passing runs too, so it discriminates nothing at the top of its range and phase 5 must read it as a goal rather than an alarm.

The correlation is a partial null, and it is computed over four jobs. Debian 12 cluster, Ubuntu 24.04 cluster, Guests and Debian 12 tier only. Node lifecycle -- the best performer in the master plan's failure table at 94% over this same window -- is uninstrumented and absent, so nothing here spans that table. Pooled, failing job-runs are much fuller (median p90 fraction 0.667 against 0.333, p < 0.001), but that comparison is confounded: three jobs share a topology and the fourth is both the fullest and the worst. Within each job, utilisation separates pass from fail not at all (p = 0.27, 0.59, 0.92, 0.74 for the cluster-wide p90 fraction; smallest p = 0.09 for the per-node maximum). What does separate them, in every job, is whether the scheduler ran out of candidates: a sufficient_idle_cpu abort occurred in 49 of 204 job-runs, and 44 of those 49 failed (90%) against 23 of 155 (15%) without one. The plan's central claim survives as "the cloud running out fails the job" and does not survive as "how full the cloud gets predicts whether the job fails".

The refusal census, four stages kept apart. Every capacity refusal in the window was at sufficient_idle_cpu: 676 drops and 14 aborts on slim-primary, 1,189 drops and 65 aborts on slim-tier, all 1,865 with the reason would exceed hard max CPUs. sufficient_idle_memory dropped 0 in 23,091 evaluations, sufficient_free_disk (disk space) 0 in 23,091, and sufficient_idle_disk (disk bandwidth, which sizing cannot address at all) 0 in 22,977. The is_hypervisor drops (7,326 on slim-primary, none on the tier) are structural, not scarcity.

D5 narrows. Memory is not a binding admission dimension: nothing, anywhere, in 23,091 evaluations. Its rationale -- one observed refusal -- does not survive the window. Its operative clause does, and now has numbers: the memory ledger is memory_max x RAM_OVERCOMMIT_RATIO at a ratio of 3.0, so the comfortable-looking fractions of ledger (p90 median 0.125 on slim-primary, 0.216 on the tier) correspond to committed vRAM at 43-76% of a slim-primary node's physical RAM and 67-101% of a tier node's. Per-node RAM is not headroom to reclaim.

D7 is closed at zero. Of 85,563 node-samples whose ledger was readable, zero fell back to cpu_hard_max and zero had no ledger at all. The cluster ledger read exactly 27.0 in all 154 slim-primary job-runs and exactly 12.0 in all 50 slim-tier ones, never moving within a run.

Survey finding 8 is real, and rare -- which corrects the step 2d brief. A capacity row above its own limit appeared in 18 of 85,563 ledgered node-samples (0.021%), all inside 2 of 204 job-runs (1.0%), both Debian 12 tier. Run 33752413862 held node 750651c4 at a committed p90 of 6.0 against a fixed limit of 3.0 (a fraction of 2.0, peaking at 2.33) for most of the run and passed; run 33948911843 peaked at 2.0 and failed. Three of the eighteen samples measured exactly zero. The brief expected this might be the single most decision-relevant number in the dataset; at one percent of job-runs it cannot explain a 76% tier failure rate, and it is not. It remains a genuine invariant violation, filed as #4087, and it remains a defect that a bigger cloud would mask.

What refused, measured from the raw bundles. Not from the committed records, which do not carry the fields. Summed committed vCPU is only 1.04x summed measured vCPU over 85,563 node-samples, and of 1,965 classifiable sufficient_idle_cpu refusals only 30.6% would have been admitted on the measurement alone. So the capacity counters are not on their own the thing that refuses: nearly seven refusals in ten would have happened from the running-domain count. Both are compared against the same cpu_schedulable x CPU_OVERCOMMIT_RATIO, so widening a node relieves both -- the master plan's conclusion is unchanged, its mechanism is sharper. In 15.1% of refusals the node measured exactly zero running vCPU.

Band bounds. Cluster-wide upper bound 0.70 defended: no slim-primary job-run reached it in 154, and 37 of 50 tier job-runs exceeded it. Cluster-wide lower bound 0.35 kept, with a question for phase 5: 104 of 154 slim-primary job-runs are below it, which is the correct finding but a useless per-run alarm, so phase 5 must decide whether that bound is evaluated per run or against a job's median across a window. Per-node upper bound 0.85 proposed, on the strongest separation in the dataset: job-runs with no capacity refusal at all (n=44) exceed 0.85 in 1 case, job-runs with at least one (n=160) exceed it in 123. No per-node lower bound is proposable -- the statistic is saturated.

One thing the window does not know, and it is not zero. Capacity guard refusals were never collected (D20) -- guard.state is not_collected on all 204 summarised records.

The second, the 2,276 of 21,517 usable samples (10.6%, present in every single job-run) whose capacity rows all read as absent at once and which are excluded from every committed-CPU figure, was written here as unknowable without D19's flag. Step 2f answered it from the shape of the data: in every one of the 204 job-runs those samples are exactly a contiguous prefix, 9 to 14 samples long, ending at the reconciler's first pass -- so it is an unpopulated table during cluster warm-up, not a failing read, and no issue is filed for it. Step 2g confirmed it, below. The 18 refusal payloads that fired with capacity_row_present false are the same window seen from the refusal path.

An error in the master plan that the data caught. Its Situation section said the sampled slim-primary failures "contain no such refusal at all" as the tier's. They do: the CPU stage dropped 676 candidates in 110 of 154 slim-primary job-runs and aborted outright in 14. slim-primary refuses work routinely and simply has other hypervisors to fall through to. Corrected in place.

Two errors in this plan that the data caught.

  • D17 says "the master plan's failure table spans six jobs". It spanned five: Ansible modules was never in it, although a merge run does build six clouds. The table has been rebuilt over the baseline window with all six rows and an instrumented column, so the confusion cannot recur. The substantive point -- that the harvest covers four of them and not Node lifecycle -- is unaffected.
  • D16's window "since 2026-08-30" is about eight hours too generous: the probe's shakenfist/actions half landed between 03:15 and 07:48 that morning, so the three earliest merge runs of the day banked bundles with no traces/ output. The harvest records them as absent rather than dropping them, which is exactly what step 2c was briefed to do, so the error cost nothing but is worth knowing before a future --since is chosen.

One smaller observation, recorded rather than acted on: the harvest writes summary: null when a bundle has no series, which discards a census that was present and readable in twelve of those thirteen records (about 100 sufficient_idle_cpu drops). Nothing in this phase needs them, and changing it would change the record schema, so it is left alone.

What the confirmation window added

Step 2g, run on 2026-09-08. Its whole job is to close the two things the retrospective window could not see, against runs that used the instrument this phase fixed: step 2a's capacity_degraded (merged to develop 2026-09-06T06:17Z) and step 2e's widened census filter (merged to shakenfist/actions 2026-09-06T00:09Z). It is a classification, not a second baseline, and nothing in What the baseline measured above is recomputed from it.

The sample. Ten merge runs asked for, 8 contributing a bundle, 32 records -- all four instrumented jobs in every one of the eight runs, every record carrying a usable series -- over 2026-09-07T10:25:26Z to 2026-09-08T01:55:33Z. Committed to docs/plans/data/ci-cloud-sizing-baseline/records-addendum.jsonl. This is the five-to-ten runs the step was scoped to and it stops there.

Blind spot 1 is closed: the guard census contains a count. guard.state is collected on all 32 records, against not_collected on all 204 baseline ones, and D20's fix therefore works in the deployed CI. What it counts is lopsided in a way worth recording:

  • 3,480 denials over 32 job-runs -- median 121 per slim-primary job-run, 134.5 per slim-tier one. In the same 32 runs the CPU pre-filter dropped 283 candidates in 3,862 evaluations and aborted 11 times, so the guard refuses a node the scheduler had already chosen an order of magnitude more often than the pre-filter drops one from the list.
  • 3,477 of the 3,480 exceed demand and nothing else, split 1,789 demand_measured_alone to 1,688 demand_estimate_tipped. The other three exceed cpus alone, two of them at the cluster stage rather than the node one.
  • Zero malformed, zero unenforced, zero with empty dimensions, zero with nothing exceeded, and no unrecognised stage or dimension name. The report's vocabulary matches what the guard actually emits, so the census is not merely present but fully classified.
  • 24 claim exceedances, one each across 24 distinct ci-claimaccount-* namespaces and on all three dimensions. These are admitted placements over an advisory claim, never refusals, and they are the claims suite exercising the path.
  • census.truncated is false on all 32, so widening the filter did not push the query into the 5000-entry limit -- the empirical form of the argument made in shakenfist/actions when the filter was widened.

A denial is not a refused create, and the master plan says so where it reports the number. Both placement walks catch CapacityAdmissionDenied, try the next candidate, and -- when nothing admitted and every refusal was demand-only, which describes 3,477 of these 3,480 -- re-walk with the demand clause waived. The count therefore mostly measures the walk absorbing a refusal, by a later candidate taking the instance or by the second pass.

What is still not counted, named so it is not mistaken for zero twice. The filter matches the denial but neither event that resolves it -- no candidate admitted and some refused on demand alone, waiving demand guard and schedule failed, every candidate refused by capacity guard. So how often the guard actually failed a create is unmeasured. Widening the filter once more is one alternation and is left to phase 3, as a Future work entry in the master plan, rather than turning this step into a third measurement window.

Blind spot 2 is closed: the warm-up prefix is an empty table. total.capacity_degraded is present on all 3,359 samples of the window and false on every one, including all 367 whose capacity rows read as absent. A failed capacity read sets it; nothing set it. The prefix pattern repeats exactly -- contiguous from sample zero in all 32 job-runs, 9 to 14 samples, 120 to 195 seconds between first and last sample, so 135 to 210 seconds of wall clock. Step 2f's inference from the baseline's shape is confirmed by direct measurement, no read-reliability issue is filed, and #4087 now rests on two independent windows.

Both halves of that are counted into the summary record rather than left in the bundles. Review of this step pointed out that the two figures which close blind spot 2 were the only claims here not recomputable from the committed dataset: they came from raw series which expire ninety days after their run, and D22 does not commit those. series.capacity_degraded_samples, series.capacity_degraded_absent_samples, series.ledger_unreadable_prefix_samples and series.ledger_unreadable_prefix_seconds were added to summary_record() at RECORD_VERSION 2, the printed report now names which of the two readings an unreadable ledger was, and records-addendum.jsonl was re-harvested to carry them. The baseline stays at version 1: its own prefix classification remains a raw-bundle figure, and the dataset README names it as one.

The window is comparable to the baseline, which is what licenses reading the two together. Cluster-wide committed-CPU p90 fraction: median 0.333 on slim-primary (n=24, max 0.481) against the baseline's 0.296-0.333 per job, and 0.792 on slim-tier (n=8, max 0.833) against 0.750. Memory dropped nothing in 3,851 evaluations, so D5's narrowing stands. Ledger provenance is zero fallbacks and zero missing ledgers over another 13,274 ledgered node-samples, so D7 stays closed at zero. Seven job-runs had a sufficient_idle_cpu abort and all seven failed; seven of the window's eight job failures had one. That is the baseline's own relationship, at a sample size which could not have established it alone.

A defect in this phase's own tooling, found by running it. tools/ci_headroom_harvest.py enumerated runs by reading until it met one created before the window, on the documented belief that the runs listing arrives newest first. The API does not promise an order and on 2026-09-08 served this workflow's merge_group runs oldest first, so 2g's first harvest wrote zero records, printed Harvesting 0 merge_group runs and exited zero -- precisely the silently half-completed harvest the tool's module docstring says it must never be. Fixed here: the window is bounded by a created=>= filter on the API call, no code path infers an order, and --limit sorts explicitly. Three tests were added, including one that pins an oldest-first listing. The baseline in records.jsonl is unaffected, and this was checked rather than assumed: asking the API today for the completed merge_group runs created in the baseline's window returns 66, which is exactly the number step 2d enumerated, so nothing was silently missed from it.

Review of that fix found three more in the same few lines, all fixed here and each pinned by a test. The created=>= boundary was formatted with strftime, which ignores tzinfo while parse_since() deliberately preserves an offset, so a --since written in this project's own timezone asked the API for a boundary ten hours later than the one requested -- and, the server-side filter running first, those hours could not be recovered. An empty enumeration could still write an empty file over a committed dataset and exit zero by any road other than the ordering one, so it now raises before --output is opened, and a set of runs which yields no records raises after it. And the window itself was not reproducible: --since 2026-09-07 --limit 10 named a different ten runs within hours of the harvest, so the tool gained --until, both boundaries are pushed down as a created=A..B range, and both datasets' README commands now name both ends. The re-harvest under the pinned window returned the same 32 records, field for field, which is also the check that the version 2 counters changed nothing else.

What this leaves for 2h. Definition of done item 10 asks the operator to confirm a real post-fix run prints a Capacity guard census with a count in it, which the 32 records above are the harvested form of. Nothing in this section changes a conclusion of step 2d; every one of them is confirmed or extended.

Risks and mitigations

  • The harvest measures a moving target. Sixty-six merge runs span a week of development, and something unrelated to sizing may shift the numbers part-way through. Mitigation: D16 requires the head SHA and creation time on every record and reporting per job rather than pooled, so a step change is visible as one; 2d is briefed to state n for every figure. This risk is smaller than the alternative's, since a prospective window has the same problem with less data.
  • Two blind spots are baked into the retrospective window. Guard refusals are uncollected (D20) and 8% of samples are discarded for an unknown reason (D19). Mitigation: 2d must report both as unknown, never as zero -- the report tool already refuses to print zero for an absent census, and 2b's record is briefed to carry the not-collected state rather than a count. Step 2g closes both against real post-fix runs before 2h can close the phase.
  • D21 pre-empts phase 5. Amending the band's form on two runs of evidence could be wrong. Mitigation: the numeric bounds still come from the harvest, 2d is explicitly briefed to narrow D21 rather than defend it if the pattern does not survive the window, and phase 5 remains the phase that turns any of it into a gate. The back brief gates this decision before 2d starts.
  • The actions half lands untested, again. Identical to phase 1's risk and identical mitigation: 2e is continue-on-error throughout, the script always exits zero, and 2g does not run until the operator confirms a real run used it.
  • The dataset commit grows without bound. Mitigation: D22 commits summary records only, not raw series; 2c is briefed to cache downloads outside the repository. If the file exceeds a megabyte, 2d should reduce what is committed rather than commit it anyway.
  • Phase 2 is tempted to propose a topology. The numbers will make a shape look obvious, and phase 3 gates phase 4 for reasons that have nothing to do with the numbers. Mitigation: D22 puts it out of scope in writing, and 2d's brief ends by saying so.

Definition of done

Falsifiable, in order:

  1. /admin/resources publishes capacity_degraded in its total block, and a unit test pins both the degraded and non-degraded cases.
  2. tools/ci_headroom_report.py exposes summary_record() and a --json option, still imports nothing outside the standard library, and still exits zero on every path. A test asserts that each figure in the printed prose equals the same figure in the record for one fixture input, so the two cannot drift.
  3. Running tools/ci_headroom_report.py on a phase 1 fixture series produces byte-identical printed output before and after 2b.
  4. tools/ci_headroom_harvest.py exists, is unit tested over a fixture bundle, and fails with a named error -- not a silent skip -- on a bundle whose artifact name is not in its topology table. The two known-uninstrumented bundles are skipped by name with a comment, and a test pins that they are skipped rather than raising.
  5. docs/plans/data/ci-cloud-sizing-baseline/ holds the harvested records and a README naming the window, the tool, and the command that reproduces it. The records are serialised compact, one per line, and the directory is under 2 MB -- a bound corrected from one megabyte once a real record was measured at 3.7 KB rather than the few hundred bytes D22 originally guessed.
  6. Every figure in the master plan's Situation section is traceable to a record in that dataset or to a cited issue. No figure in it is described as hand-collected.
  7. The master plan no longer says the slim-tier ledger is "12 by derivation, 10 by observation", and phase 0's D7 is marked closed with the evidence. (Done in the planning commit; re-check rather than redo.)
  8. The baseline states, for guard refusals and for ledger-unreadable samples, that the retrospective window does not know -- and neither is reported as zero anywhere in it.
  9. Phase 0's D5 is either confirmed or narrowed against the window's memory figures, with n stated, rather than left as the single-run judgement it is today.
  10. The census filter in shakenfist/actions matches both guard messages, and a real post-fix run's Capacity guard census section contains a count rather than the "not collected" notice. The operator confirms this; it cannot be checked from a worktree.
  11. No step added or changed by this phase can fail a CI job: every actions step remains continue-on-error and every script still exits zero, verified by reading the diff.
  12. python3 tools/check-plan-status.py passes, and pre-commit run --all-files passes in the main repository.

What phase 3 inherits

  • A distribution, so "the cluster really is full" is a measured claim rather than a sampled one, and the boundary phase 3 must write a test against is a known number.
  • A per-stage refusal census over many runs, which is the corpus phase 3's inventory of signatures should be checked against -- phase 0 built that inventory from triage history, and this is the first chance to see whether the frequencies match.
  • A working capacity guard census, from D20, so a test which fills a cluster to its ledger can assert on the guard's own events and not only on the pre-filter's -- proven working, with 3,480 denials counted over 32 job-runs in What the confirmation window added. Phase 3 should widen the filter by the two events which say what became of a denial before writing that test; the master plan's Future work carries the entry.
  • An answer on whether memory binds, which decides whether phase 3's saturation test needs a memory dimension at all.
  • The knowledge that the cluster-wide fraction and the per-node maximum can disagree by a factor of two, so a saturation test which fills "the cluster" must say which of the two it means.

Back brief

Before executing any step of this plan, back brief the operator on the understanding of it, and in particular on:

  • D16, which replaces the master plan's waiting window with a retrospective harvest. This is the decision that changes what the phase is.
  • D21, which amends D3's band form on two runs of evidence. This is the one a reviewer is most likely to disagree with, and it should be agreed before 2d starts writing conclusions against it -- rewriting the baseline afterwards is expensive.
  • D22's dataset commit, whose shape and location should be agreed before 2d generates it.

Step 2e lands in shakenfist/actions, where it cannot be tested before it merges and where only the operator can push. Step 2g cannot start until 2a and 2e have merged and real merge runs have used them, so the phase has a hard pause in the middle that is not a stall.

📝 Report an issue with this page