A capacity refusal is transient

Prompt

Before responding to questions or discussion points in this document, explore the shakenfist codebase thoroughly. Read relevant source files, understand existing patterns (object lifecycle, state machines, MariaDB storage via the three-layer direct/gRPC/public pattern, Pydantic schemas, daemon architecture, operation queue system, event logging), and ground your answers in what the code actually does today. Do not speculate about the codebase when you could read it instead. Where a question touches on external concepts (KVM/libvirt, VXLAN networking, MariaDB/Galera, gRPC/protobuf), research as needed to give a confident answer. Flag any uncertainty explicitly rather than guessing.

This plan spans three repositories: this one (the scheduler, the cluster and resources daemons, and the shakenfist_ci functional suite), client-python (shakenfist_client.apiclient, which is what the suite and every operator script calls), and, for one step only, shakenfist/actions (where the reusable smoke-cluster workflow collects the per-run summary this plan adds). It is a sibling of Right-size the CI test clouds and Atomic scheduling via reservations, and deliberately does not own anything either of those already owns: topology shape belongs to the first, the admission ledger and claims to the second. What is left, and what this plan is, is the piece both of them explicitly deferred -- that nothing in the system treats "no capacity right now" as the transient condition it is.

Consult ARCHITECTURE.md for the system architecture overview, object types, and daemon structure. Consult CLAUDE.md for build commands, project conventions, and database access patterns. Consult GOALS.md for current development priorities. Key references inside the repo include shakenfist/scheduler.py (_has_sufficient_cpu, the pre-filter that raises the refusal this plan is about), shakenfist/external_api/instance.py (the two 507 branches of POST /instances), shakenfist/daemons/cluster/main.py (_force_capacity_reconcile_if_unguarded and the anchored reconcile), shakenfist/daemons/resources/main.py (the 60 s metrics cadence that measured_cpus comes from), shakenfist/deploy/shakenfist_ci/base.py and retries.py (the suite's await and retry helpers), and shakenfist/deploy/shakenfist_ci/load_budget.py (HARNESS_DRIVEN_PAIRS, which any new suite-side polling must be declared in).

Plan file conventions (shared block; do not edit -- the canonical copy lives in shakenfist/development at templates/shared-blocks/plan-file-conventions.md):

• All planning documents live in docs/plans/.
• Detailed planning gets one plan file per phase. Phase files are named for their master plan, sit in the same directory as it, and append -phase-NN-descriptive before the .md extension.
• The master plan tracks its phases in a table under its Execution section:

Phase	Plan	Status
1. Schema migration	PLAN-thing-phase-01-schema.md	Not started
2. Public API	PLAN-thing-phase-02-api.md	Not started

• One commit per logical change, and at minimum one commit per phase. Unrelated changes are not batched into a single commit. Each commit is self-contained: it builds, passes tests, and has a message explaining what changed and why.

Situation

The cost

Issue #3772 is the signature in essentially every failing merge group. Over the sixteen days to 2026-09-08 the Functional tests workflow's merge_group runs split 108 failures, 45 successes, 14 cancelled. Of the thirty most recent failing runs, twenty-four failed the Debian 12 tier job (the slim-tier topology). Of eighteen of those runs whose failed-job logs were read, all eighteen contain

shakenfist_client.apiclient.InsufficientResourcesException: ('API request failed', 'POST',
'http://localhost:13000/instances', 507,
'{"error": "No nodes remaining at scheduling stage sufficient_idle_cpu", "status": 507}')

and in fourteen a known victim of that refusal is the test that failed: test_network_plumbing_lifecycle (8), test_disappearing_source_instance (3), test_stray_torn_down_while_a_hosted_network_survives (3), then test_artifact_show, test_lifecycle_power_cycle and the affinity tests. The next-largest cause, the database load-budget check, appeared in seven of the eighteen -- and in six of those a 507 was in the same run, so fixing the budget family alone would not have turned one of them green.

What the refusal actually is

The evidence below comes from the full CI bundles -- the 15 s headroom series, the Loki refusal census and every node's journalctl -u 'sf-*' export -- of six failing merge runs after PR #4106 landed (34163288637, 34171977552, 34178278720, 34119030297, 34125365386 on slim-tier; 34168326220 on slim-primary), two before it (33991296717, 33948911843), and the 50 usable slim-tier records of the sizing plan's baseline dataset. The reproduction recipe and the analysis script are under docs/plans/data/transient-capacity-refusals/. +Ns is seconds after the Run functional tests step started.

Every sufficient_idle_cpu abort in the six post-#4106 runs was a force_placement single-candidate create onto a node whose capacity ledger was genuinely at its limit, while the cluster as a whole held 3-9 of its 12 vCPU. The ledger reconstructed from the instance placed and instance placement released events in every node's journal matched the pre-filter's committed_cpus exactly in all ten cases. It is not over-counting, not a race, and not the warm-up window.

run	topology	first capacity row	cpu abort(s)	after first row	forced onto	measured / committed / limit
34163288637	tier	+181s	+309s	+128s	sf1	2 / 3 / 3
34171977552	tier	+166s	+512s	+346s	primary	3 / 3 / 3
34178278720	tier	+181s	+200, +201, +202, +314s	+20..+133s	sf1, sf1, sf2, sf1	0/3/3, 0/3/3, 1/6/6, 3/3/3
34119030297	tier	+165s	+481s	+316s	sf1	6 / 0 / 3
34125365386	tier	+166s	+696s	+530s	sf2	3 / 6 / 6
34168326220	primary	+166s	+824s	+658s	sf1	3 / 3 / 3
33991296717 (pre)	tier	+166s	+519, +542, +565, +725s	+354..+559s	none: all three nodes	sf2 6/6, sf1 3/3, primary 3/3
33948911843 (pre)	tier	+181s	+176, +177, +523s	-5s, -4s, +342s	primary, primary, primary	0 / 6 / 3, 0/6/3, 3/3/3

(The +200 s trio and the +176 s pair are test_duplicate_network_work_is_coalesced, which pins a burst of six across the hypervisors and tolerates its own create errors; the test that failed each run was a later one.)

Three things follow from the table.

1. The binding constraint is the 3 vCPU ledger on the two infra hypervisors. primary (hypervisor, network node and database node) and sf1 (hypervisor and database node) are 4-thread VMs carrying the deploy-time reservation of 4 threads (examples/_shared/site.yml, (1 + infra_role) * 2), so cpu_schedulable = max(1, 4 - 4) = 1 (shakenfist/daemons/resources/main.py:126) and limit_cpus = floor(1 x 3.0) = 3 (shakenfist/mariadb.py, _derive_cpu_memory_limits). sf2 gets 6. Cluster 12, confirmed at exactly 12.0 in all 50 baseline records.
2. The victim tests pick those nodes deterministically. test_network_plumbing_lifecycle takes the first two non-network hypervisors (sf1, sf2); test_disappearing_source_instance the first hypervisor; test_stray_torn_down_while_a_hosted_network_survives the first non-network hypervisor (sf1); test_artifact_show its first instance's node. Five stestr workers queue their pinned creates on a node three 1-vCPU instances fill, and the instances that fill it are still fetching images (measured 0, committed 3) when the next one is refused.
3. The refusals are not concentrated in the warm-up. They land between +176 s and +824 s, through the first fourteen minutes of a 25-35 minute step, after the ledger rows exist. In the baseline, the ledger-3 nodes sat at 100% of ledger at p90 in 65% of node-records (peak at or above 1.0 in 96%); 33 of the 37 slim-tier failures had at least one abort; slim-tier passed 13 of 50.

Per-node saturation across the six post-#4106 runs (max(measured, committed) >= limit):

node (ledger)	share of the run at ledger	longest stretch
sf1 (3)	7-23%	90-330 s
primary (3)	1-12%	15-255 s
sf2 (6)	0-15%	0-240 s

Three real mechanisms that are not the driver

The warm-up window is still open. Issue #4087 found that scheduler_node_capacity has no rows until the reconciler's first pass, so every early placement takes P7's fail-open branch and the first pass then writes the accumulated ground truth onto a row whose limit it exceeds. PR #4106 anchored the reconcile to election and added a one-shot _force_capacity_reconcile_if_unguarded() (shakenfist/daemons/cluster/main.py:741-778, called at :879). In all six post-#4106 runs that one-shot fired 130-150 s before the test step, and the pass it forced logged nodes=0 nodes_added=0: no hypervisor had published metrics yet. The table then stayed empty until the five-minute cadence created rows at +155..+181 s, exactly the 135-210 s the baseline measured before the fix. Ten to seventeen placements per run were admitted node_not_sized, and the first real pass recorded drift_cpus of 2-8 (34178278720: sf1 +6 on a limit of 3). The bold rows above are this residue. It is real, it is the whole of the sub-three-minute exposure, and it is not what fails the runs.

Measurement lags a delete burst by a metrics period. The 34119030297 refusal was measured 6 / committed 0 / limit 3: six warm-up instances had been deleted and released 32 s earlier and the ledger was correctly zero, but cpu_total_instance_vcpus is republished every 60 s (resources/main.py:648, 704) and still counted six running domains. The next sample read zero. Because _has_sufficient_cpu() charges max(measured, committed), a node refuses forced creates for up to a minute after a teardown even though it is empty. The same rule is why "release the ledger earlier in Instance.delete()" -- a proposal the research considered -- would not help: the measured side keeps charging the node until the domain is actually gone, and the journals show no deleted instance holding ledger at any refusal.

The demand guard is pure overhead on this topology. 114-140 schedule candidate refused by capacity guard events per run, 100% demand-only: the D13 bound is 0.75 x cpu_schedulable, which is 0.75 on a 1-thread node against a cpu_load_1 near 3. Every one was waived on the second walk, so roughly 40% of creates pay two guarded transactions at the moment the cluster is busiest. It never produced a 507 in these runs. It is owned by the scheduler-reservations plan and recorded there.

What has been tried

• #3724 added the committed-vCPU ledger so placed-but-unbooted instances are charged immediately. Admission became more accurate; an accurate "no" is still a failed test. Three recurrences followed and the six per-test issues were absorbed into #3772.
• #3722 reordered the load-shedding filters below affinity. Cannot help: the node is removed at admission before ranking.
• #4106 closed #4087 in principle and, as above, not in CI.
• #3813 made the demand guard satisfiable; the waiver rate fell from 62% to 4%. Not a 507 cause.
• #3907 and #3565 were closed test-side, by tolerating the transient refusal and by skipping when the candidate set had collapsed. Both are precedents for the shape of this plan and neither generalised.

What this is not

Claims are not the answer to a forced placement. A namespace claim is cluster-wide and carries no node affinity (scheduler-reservations D14), shakenfist/scheduler.py never reads namespace_claims, and the stage that refuses is per-node. Per-test namespaces holding claims would starve one another's unclaimed pool, and phase 5's hard ceiling caps the holder rather than helping it. The one legitimate claim in CI is the conductor holding one per run against other merge groups on the under-cloud, which is scheduler-reservations phase 4c. Node-scoped claims are considered under open question 5 below and declined for now.

Mission and problem statement

Make a capacity refusal something a caller can wait out rather than a failure it has to report, without hiding the refusal from the people who need to see it. Concretely:

• A cluster's first guarded placement should happen within seconds of its hypervisors publishing metrics, not five minutes after its cluster daemon started.
• A node should stop charging for instances that no longer exist within seconds of their deletion, not within a metrics period.
• The functional suite should treat a 507 at create_instance as "not yet" -- waiting, informed by the cluster's own published headroom, for the node it needs -- and should say, per test and per run, how long it waited, so that a topology that makes the suite wait is visible rather than merely slower.
• The API should tell a client that a refusal is transient and when to try again, and the Python client should be able to act on that when asked to.
• Whether the server itself should queue a create until it fits is decided from measurement, not argued from first principles, and the decision is written down either way.

The merge-queue outcome this is measured by is the Debian 12 tier job's pass rate, which is 24% in the baseline and should be comparable to the other cluster jobs (78-96%) once this plan and the sizing plan's phase 4 have both landed.

Open questions

These were settled at planning time from the research recorded in the Situation section; the reasoning is kept so that a later reader can disagree with it on the evidence rather than re-deriving it.

1. Where does the retry live?

In the suite first, in the client second, in the server only if the data says so. The scheduler-reservations plan left this undecided ("the client SDK, the CI base class, or server-side admission queueing") and deliberately held it until #3772 had soak data from a develop carrying atomic admission. That soak data now exists -- the phase 2 baseline and the journals above -- and it says the refusals are correct against the ledger. A suite-side wait is the smallest change that stops the bleeding, it is where retry_while_transient already lives, and it is the only place a wait can be informed: GET /admin/resources publishes per node cpu_available = cpu_hard_max - max(measured, committed), which is the pre-filter's own arithmetic, so a pinned test can wait for the node it needs rather than for the cluster. The client change is next because it is what makes the behaviour available to operators and to the downstream repositories' suites. A server-side queue reverses scheduler-reservations D8 and is phase 5's decision.

2. Does a retry hide the problem?

Only if it is silent, so it will not be. Both sibling plans say a retry would mask whether a bigger cloud or atomic admission actually changed the failure rate. The wrapper in phase 2 therefore records every wait as a test detail (how long, on which node, what the headroom looked like) and the run publishes a summary -- total seconds waited, waits per test, longest wait -- through the same bundle the headroom probe uses, so the sizing plan's phase 5 guardrails can warn on it. A cloud that makes the suite wait becomes a number in every run instead of a flake in some. The sizing plan's phase 3 saturation tests, which assert what a full cluster does today, call the raw client and are exempt from the wrapper by an explicit marker.

3. Why is allocation less reliable in the first minutes, and can the window be shortened?

Because the one-shot fires before there is anything to reconcile; yes, from 135-210 s to roughly a minute. _force_capacity_reconcile_if_unguarded() runs once, on the election path, and tests if rows:. On a fresh cluster the cluster daemon wins its election within 2.5-7.5 s of starting, before sf-resources on any hypervisor has published -- the pass finds nothing, and nothing re-checks. Phase 1 makes the check a property of the elected loop rather than of election: compare the set of active hypervisors that have published metrics against the set with a capacity row, and make the reconcile due when they differ. That also closes the narrower hole

4106 left -- a node that publishes late, or whose row the

reconciler removed on stale metrics, admits unguarded for up to five minutes today.

Note what bounds the improvement, because an earlier draft of this section claimed seconds. The elected loop polls every ELECTED_LOOP_POLL_SECONDS = 5 (shakenfist/daemons/cluster/main.py:63) but its maintenance body sits behind if now - last_loop_run >= 60 (:913), and _run_due_scheduled_jobs() is inside that gate (:918). Marking the job due does not run it any sooner than the next 60 s tick, so a check placed inside the gate closes the window to about a minute. Closing it faster means running the comparison outside the gate on the 5 s poll, which is a fixed-rate database read and is not free. Phase 1 decides between those two; see its plan.

4. Why does a node refuse for a minute after its instances are deleted?

Because measurement is a 60 s poll of libvirt and the pre-filter charges the larger of measurement and ledger. The rule is right: it is what makes the ledger safe against instances the reconciler has not yet counted. What is wrong is the cadence. Phase 3 has sf-resources notice a change in the running-domain set cheaply (a listDomainsID every few seconds costs one local libvirt call and touches no database -- note that get_all_domains() is not that call, it is listDomainsID plus a lookup per domain) and publish immediately when it changes, keeping the full 60 s publish for everything else. That is activity-coupled rather than fixed-rate, so it does not move the database load budget's idle figure, and it is declared in database_load_budget.yaml as such.

5. Should there be node-scoped claims?

Not now, and not for this problem. The question was raised at planning time because a claim "on hypervisor X for N vCPU" is the shape a pinned test would want: hold the reservation, then create against it, with no window between "the node has room" and "my create landed" for another worker to take it. Against that:

• For CI it buys one thing over an informed wait -- closing that window -- and the window is small at concurrency 5. Every other property (waiting for the right node, bounded by a deadline, visible in the run) the phase 2 wrapper has already.
• It is a second ledger dimension on every scheduler_node_capacity row (claimed_cpus beside used_cpus, with the unclaimed admission guarded against limit - claimed), which the guarded UPDATE, the reconciler, the cluster singleton's migration on claim create and delete, and the pre-filter all have to learn. That is scheduler-reservations phases 3 and 4 again, one level down, while phase 5 has not decided what enforcement of the existing claims even means.
• The pre-filter is claim-blind today; a node claim only works if _has_sufficient_cpu() consults it, which makes the pre-filter a ledger reader in a way D1 chose not to.

Where it would earn its place is an operator need, not a test need: evacuating or draining a node (#1364) has to know there is room on the destinations before it starts moving instances, and a live migration wants the same guarantee. When that lifecycle is built, a per-node reservation is the primitive it needs, and it should be designed then, against the claim machinery as it stands after phase 5. Recorded under Future work in the scheduler-reservations plan so it is not lost.

6. Should the demand guard be changed here?

No; it is scheduler-reservations' and is recorded there. The bound of 0.75 x cpu_schedulable cannot be met on a 1- or 2-thread node under any real load, so on the CI topologies the first walk never admits and the waiver walk always does. Rescaling or waiving it below some cpu_schedulable is a one-line change to the demand clause, but it is phase 4a's clause and the load it saves is the scheduler-reservations plan's to measure. This plan cites the evidence and moves on.

7. Should the topology change here?

No; it is the sizing plan's phase 4, and the evidence here sharpens what that phase must choose. Any shape that leaves an infra hypervisor at limit_cpus 3 leaves this failure in place, because the pinned tests select those nodes. The sizing plan's candidate "tier as 3 x 6 vCPU" gives primary and sf1 a ledger of 6 each and sf2 12, which is the smallest shape in its table that changes the number that binds. The sizing plan is updated to say so.

8. Should the server queue a create that does not fit?

Not answered yet. Phase 5 took its first reading, over 26 qualifying merge_group runs on the reshaped slim-tier, and the window was not readable: 96 of its 104 instrumented bundles carry no capacity-wait trace at all, an unknown fraction of 92.3% against the 25% ceiling phase 5's pre-registered rule sets for calling a window readable. The cause is not lost plumbing -- every absent bundle carries a fully populated bundle/traces/ -- but that the trace file is created on first write and so cannot express "nothing was refused" (#4337). That is the rule's unreadable case, which is deliberately not one of its three outcomes and deliberately does not resolve to Abandon, so the phase extends its window once rather than deciding on the eight readable units. The full reading, the arithmetic for all three outcome clauses under both candidate denominators, and what the second reading owes before it is taken are in phase 5's Outcome.

One thing the second reading owed has since been paid. The rule is amended, as D43, written before any of the second window was read: now that #4337 makes zero expressible, an empty trace is an observation of zero waits and enters the denominator, while absent and unparseable stay unknown. That is what unpins the two clauses the first reading found structurally stuck at 100%, and it gives the rule back a path to Abandon that it did not have. Nothing else in the rule moves.

What the eight readable waits say, held loosely because they are 7.7% of the window: waits from 0.036 s to 270.858 s against the suite's 420 s deadline, none of which reached it; all eight pinned creates, which means the fairness assertion below is still unevidenced rather than confirmed; seven bound on cpus. On the reshaped slim-tier alone the longest was 180.490 s, below the 210 s line the rule draws. None of that decides anything.

The arguments, which the reading did not disturb. The machinery exists -- BaseClusterOperation.defer_with_backoff() already re-enqueues with a delay for artifact fetches and network operations -- and a 202 with the instance held in initial (or a new scheduling state) until placement succeeds or a deadline passes looks like a contained change. Against it: it is exactly the "hold-until-fittable" that scheduler-reservations D8 rejected as queue-state surface the project does not want; it has no fairness model (a waiting 4-vCPU create starves behind a stream of 1-vCPU ones, and a pinned create starves worst -- an assertion no data supports yet, because every refusal phase 5 could read was pinned and there was no comparison arm); waiting instances hold IPAM allocations, so a CPU shortage can become an address shortage; and the client's create-and-await path has a ceiling that bounds any useful deadline. Which way those arguments fall is decided by phase 5's numeric rule on the second reading, not by "short and few" versus "long or many" -- that informal test is what the phase plan replaced with thresholds fixed before the data existed.

Phase 5's survey corrected four of the sentences above, and the corrections move the arithmetic rather than the prose. See its What the survey found for the full readings.

• The client method is await_instance_create(), spelled without a leading underscore, and its default ceiling is 600 s, not 900 (client-python/shakenfist_client/apiclient.py:1797). Its docstring also warns that create_instance(timeout=...) (:737-749) bounds the whole call, so a caller that does not pass timeout=0 runs "two independent budgets on the same condition, run back to back" -- a server-side hold would be a third.
• defer_with_backoff()'s default schedule is delays=(15, 30, 60) (shakenfist/operations/baseoperation.py:708) -- three defers totalling 105 s, after which it returns False and the caller must error the operation out. All three waits phase 2 actually measured (190.5 s, 160.4 s, 140.6 s) are longer than that entire budget, so "the machinery exists" is true but would give up before the shortest recorded wait had cleared. A queue needs its own schedule; the re-enqueue is not the hard part.
• The IPAM concern is confirmed and is structural rather than incidental. _netdesc_allocate_address() reserves the address and is called at external_api/instance.py:1027; find_candidates() runs at :1049, twenty-two lines later. The only thing that releases those addresses today is the refusal itself -- every scheduling refusal branch calls enqueue_delete_due_error() (:1062, :1069, :1082, :1177). A queue that holds the instance instead of deleting it holds its addresses for the whole hold, by construction.
• POST /instances has four 507 branches, not two. The two phase 4 routed through capacity_error() are at :1076 and :1184; the other two are exceptions.CongestedNetwork at :477 and :509 and return a bare sf_api.error(507, ...) with no Retry-After, stage or transient field. So a queue that converted CPU pressure into address pressure would convert a refusal a client is told to retry into one it is told nothing about.

9. What should the API say?

That it is transient, and when to try again. Both scheduling 507 bodies from POST /instances (external_api/instance.py:944 for the pre-filter, :1017 for the guard) are bare strings. Phase 4 adds a Retry-After header and a machine-readable stage and transient: true to the error body, so a client does not have to parse prose to know which refusal it got. The honest hint is a fixed conservative constant (15 s, matching defer()'s default) rather than a computed one: the server has no pending-release horizon and a number that implies knowledge it lacks is worse than one that does not. The client's opt-in retry reuses the shape of its existing 406 loop and is bounded by the same deadline. The affinity 409 is deliberately ordered first in the handler and must never be retried.

Execution

Plan status vocabulary (shared block; do not edit -- the canonical copy lives in shakenfist/development at templates/shared-blocks/plan-status-vocabulary.md):

A status cell -- in the master plan's own Execution phase table, and in the row docs/plans/index.md carries for the plan -- holds exactly one of these terms and nothing else:

• Proposed -- written down as a concept, not yet scheduled.
• Not started -- scheduled, but no work has begun.
• In progress -- work has begun and has not finished.
• Blocked -- cannot proceed until something outside the plan changes. Say what, in the plan.
• Complete -- the work is done.
• Abandoned -- deliberately dropped without being done.
• Superseded -- replaced by another plan, which the plan names.

The term is the whole cell. No dates, no phase arithmetic, no parenthetical qualifiers, no summary of what happened: a status is read to decide whether a plan still wants attention, and prose in that column has repeatedly grown until it could no longer be read either by a person scanning the table or by tooling. Detail belongs in the plan file, and a one-line summary belongs in the index's own Intent column.

Matching is case-insensitive, so In Progress is accepted, but the spelling above is the one to write.

In this project

The same term is written twice: once in the phase table below, and once in the row this plan carries in docs/plans/index.md. Keep them in step -- the index row is the whole-plan status, so it only reaches Complete once every phase has been completed, abandoned or superseded.

Phase	Plan	Status	Merged
1. Close the warm-up window: reconcile when a hypervisor has metrics and no capacity row	PLAN-transient-capacity-refusals-phase-01-warm-up.md	Complete	7cc93750d (#4147), e20dd7d4b (#4153)
2. The suite waits, and says so: an informed create_instance wrapper and a per-run wait summary	PLAN-transient-capacity-refusals-phase-02-suite-wait.md	Complete	5ad9651ee (#4166), 2c6206941 (#4187)
3. Publish metrics when the running-domain set changes	PLAN-transient-capacity-refusals-phase-03-metrics-on-change.md	Complete	03cd7be3a (#4200)
4. Retry-After and a machine-readable transient refusal, with an opt-in client retry	PLAN-transient-capacity-refusals-phase-04-retry-after.md	Complete	565e36e6e (#4241), client-python 74d6e129b (client-python#399)
5. Decide on server-side queued placement from the phase 2 data	PLAN-transient-capacity-refusals-phase-05-queue-decision.md	In progress	—
6. Documentation and close-out	PLAN-transient-capacity-refusals-phase-06-docs.md	Not started	—
7. Push audit	PLAN-transient-capacity-refusals-phase-07-push-audit.md	Not started	—

The Merged column records what put each phase on develop: the merge commit of its pull request, or an explicit first..last range where the phase landed directly. It is filled in as each phase lands, so — means the phase has not landed yet, even where its phase plan is already written and linked above. Phase 1 took two: the implementation, and the close-out which recorded its measurement. Phase 2 took the same two.

Phases 1, 2 and 3 are independent of one another and can run in parallel. Phase 4 follows 2, because the client retry should match the semantics the suite has already proven. Phase 5 needs phase 2 to have reported over a window of merge runs after the sizing plan's phase 4 has reshaped slim-tier; until then its data would be measuring the wrong cloud. Phase 6 is the last of the implementation phases; phase 7 is the push audit, which reads all of them.

The ordering against the sibling plans: the sizing plan's phase 3 (saturation coverage) does not gate any phase here, because none of them changes what a full cluster does -- but its phase 4 (reshape) should land before phase 5 here reads its numbers. Nothing here touches the demand guard, the pre-filters, claim enforcement or the guarded UPDATE, all of which are scheduler-reservations' (D11, phase 5).

Phase 1 -- Close the warm-up window

Make _force_capacity_reconcile_if_unguarded() a check the elected loop repeats rather than a one-shot at election (shakenfist/daemons/cluster/main.py:741, called once at :879). The check itself is the same comparison either way: if any active hypervisor has published metrics and has no capacity row, make the reconcile due now. Keep the distinction between a degraded read and an empty result that the existing code is careful about (rows is empty for both; only degraded says which).

The decision this phase owns is where the check runs, and the master plan does not pre-empt it. Inside the elected loop's 60 s maintenance gate (:913) the check costs nothing new -- it rides a pass that already reads the database -- and closes the warm-up window to about a minute. Outside the gate, on the ELECTED_LOOP_POLL_SECONDS = 5 poll, it closes the window to seconds but adds a fixed-rate ~0.2/s read that needs a cluster_base_qps entry in shakenfist/data/database_load_budget.yaml, or test_no_unbudgeted_fixed_rate_database_polling fails. The phase plan picks one and says why; both are defensible and the 60 s version is the smaller change.

Be accurate about the stability gate rather than repeating the elected loop's shorthand comment. cluster_stable() (shakenfist/daemons/daemon.py:377) compares object versions across nodes and reads no metric freshness at all; it catches a just-restarted cluster only incidentally, because no node has recorded a version yet and minimum is inf. The pass still belongs behind it -- issue #4087's second correction explains why a pass with no fresh metrics deletes rows -- but the protection is a side effect of the version check, not a freshness check, and the phase plan should not assume otherwise.

Prove it two ways. A unit test in shakenfist/tests/ drives the elected loop with a fake capacity table and a fake node roster and asserts the reconcile becomes due on the iteration a hypervisor first appears without a row, and not on later iterations where every hypervisor has one. A functional assertion in the CI harness reads the headroom probe's own series and requires that cpu_committed_row_present is true for every hypervisor before the first instance placed event of the run -- the field already exists end to end (published at shakenfist/scheduler.py:1073, harvested by tools/ci_headroom_harvest.py, reported by tools/ci_headroom_report.py:319) and the baseline dataset already carries it, so the assertion's premise can be checked against it before it is written. That assertion also retires a skipTest: cluster_ci_tests/test_nodes.py:136 currently skips when a node has no capacity row, which is exactly the condition this phase makes impossible after start-up. Remove the skip in the same change rather than leaving unreachable code behind it. Phase 1 also comments on #4087 with the finding and closes it. The issue is still open -- #4106 never closed it -- so there is nothing to reopen.

Small, server-side, one file plus tests. Plan at high effort: the placement decision above, the interaction with the stability gate and the degraded-read distinction are the kind of thing a light brief gets wrong.

Phase 2 -- The suite waits, and says so

Add BaseTestCase.create_instance() to shakenfist/deploy/shakenfist_ci/base.py, route every raw test_client.create_instance(...) call site in shakenfist/deploy/shakenfist_ci/ through it (there are 117 call sites across 41 files; a list is a mechanical grep), and add a unit test in shakenfist/tests/ that walks the suite's source with ast and fails on any raw call outside an explicit allowlist marker, the way test_ci_claims_headroom.py already asserts call sites by name.

Amended by the phase 2 survey: this section originally named only cluster_ci_tests/ and guest_ci_tests/, which hold 88 of the calls. The 115 it quoted was a count over the whole of shakenfist/deploy/shakenfist_ci/, which today is 117 across 41 files. The difference is smoke_ci_tests/ (27 calls in 7 files), base.py:1227 and database_tier.py:275. The smoke tier belongs in scope: it runs on the smallest cloud, where one sibling instance is the largest fraction of the cluster.

The wrapper: on InsufficientResourcesException, if the deadline (420 s, the claims suite's CLUSTER_HEADROOM_WAIT) has not passed, emit a tracing event, then wait informed: poll system_client.get_cluster_resources() at 10 s and proceed when per_node[target]['cpu_available'] >= cpus for a force_placement create, or total['cpu_available'] >= cpus otherwise, falling back to a blind sleep when total['capacity_degraded'] is set. Re-create with a fresh uniquified name, because the refused instance is enqueue_delete_due_error'd and its name is not reusable synchronously. Reuse retries.retry_while_transient by converting the exception to a (status, body) pair so the loop stays unit-testable against a fake clock. Every wait is attached to the test as a detail: seconds waited, the node waited for, and the per_node headroom at the first refusal and at admission.

Amended by the phase 2 survey: both halves of the predicate above are wrong, and phase 2's D8 replaces them. per_node[n]['cpu_available'] is derived from the live overcommit arithmetic (scheduler.py:1086-1088), not from the capacity row's limit_cpus that admission actually refuses against -- the endpoint publishes both precisely so a reader can see them disagree. And total['cpu_available'] is a sum across nodes (:1091-1093) while an instance must fit on one, so it is satisfied by a cluster with a spare thread on each of ten nodes. retry_while_transient is also not reusable as written: it re-issues the request every 10 s, which here means a fresh refused create -- and each refused create builds an instance and error-deletes it -- and its module deliberately imports nothing from the suite so the unit tests can load it by path.

The run summary: total seconds waited, number of waits, longest wait and its test, written to the bundle beside the headroom probe's output and printed in the job log. The collection step is in shakenfist/actions (the reusable smoke-cluster workflow), so this phase carries the same operator-push obligation the sizing plan's phase 1 did, and the summary is designed so that the sizing plan's phase 5 guardrail can read it.

Amended by the phase 2 survey: only the job-log print carries that obligation. smoke-cluster.yml:212 creates /srv/ci/traces on the primary and chowns it to the base-image user, and the "Gather logs" step at :550 scp's the whole directory into the artifact bundle; the suite runs on the primary as that user. So a file the suite writes there reaches the bundle with no change in shakenfist/actions at all, and the report can be run over a downloaded bundle -- which is how phase 1 ultimately verified its own definition-of-done item 9. Phase 2's D14 puts the cross-repo change last and lets nothing depend on it.

Budget the poll. GET /admin/resources reads GetNodeMetrics from the api caller, which is already in HARNESS_DRIVEN_PAIRS (load_budget.py:309-323) -- but that exemption's prose names the headroom probe as the producer and test_the_suite_still_probes_cluster_headroom holds it up. Extend the comment to name the wrapper as a second, activity-coupled producer (it polls only while a create is being retried), or the load-budget test's premise rots silently.

Two test-side changes ride along because they are cheap and reduce the number of pinned creates: test_commandline_artifacts.py's second instance and test_imagefetch.py's first are pinned for convenience rather than for the assertion, and can target inst1['node'] only where co-location is actually load-bearing. And test_system_namespace.py subclasses BaseTestCase rather than the namespaced base, so a failure between its inline create and delete strands a charged instance in the system namespace for the rest of the run; give it an addCleanup.

Amended by the phase 2 survey: test_commandline_artifacts.py already has the shape asked for -- :182-190 pins the second instance to inst1['node'] and the first is not pinned at all -- so there is nothing to change there. test_imagefetch.py has four pinned creates, not one. In its first test both (:130, :152) target an arbitrary first hypervisor, and since the source URL is deleted between them the second one's co-location is load-bearing: the first's pin comes out, the second's becomes inst1['node']. Its second test pins first and second deliberately, to place the second instance on a node which has not seen the image, and must be left alone. The test_system_namespace.py observation is correct as written.

The sizing plan's phase 3 saturation tests must call the raw client and assert the refusal; the allowlist marker exists for them.

Plan at high effort. The wrapper is straightforward; the AST guard, the load-budget declaration and the bundle plumbing across two repositories are where a light brief goes wrong.

Phase 3 -- Publish metrics when the running-domain set changes

In shakenfist/daemons/resources/main.py, beside the 60 s publish, poll the set of active libvirt domain ids every few seconds (no database access) and, when that set differs from what was last published, publish immediately. Keep the 60 s full publish unchanged, resetting its clock on any publish. Declare the changed load in shakenfist/data/database_load_budget.yaml, and add a unit test that a domain disappearing between polls produces a publish before the 60 s tick.

This is what makes "the node is empty" true within seconds of a teardown rather than within a minute, and it is the only change in this plan outside the scheduler, the API and the test suite. Plan at medium effort; the pattern is the existing loop.

Three things this section originally got wrong, corrected when phase 3 was planned and set out in full under What the survey found there. A publish is not a single write: it is a fifteen round-trip _get_stats() sweep, twelve of them queue-depth reads, so the cadence change has a bill worth bounding. There is no UpsertNodeMetrics entry in the budget file to amend -- entries must be added -- and activity_coupled does not refine the model, it switches enforcement off (load_budget.py:443-450), so it is applied only to the pairs the publish path actually reaches. And a vCPU total is not a second trigger worth watching: with no CPU hotplug it can only move when the domain set does.

Phase 4 -- Retry-After and a machine-readable transient refusal

Server: on the two scheduling 507 branches of POST /instances (external_api/instance.py:944, the filter pre-check, and :1017, every candidate refused by the capacity guard), set Retry-After: 15 and extend the error body with stage and transient: true. POST /instances has four 507 branches, not two: the other two are the CongestedNetwork clauses at :428 and :451, and phase 4's D29 leaves those bare because an exhausted address pool has a different and much longer horizon. Phase 4's review round refined this further: the filter branch is not one fact either. A structural stage such as cpu_max_per_instance gets a 507 carrying its stage but transient: false and no header, because no wait clears it. See phase 4's D35. sf_api.error() returns a bare flask.Response, so the header is set on the returned object -- but note it is shakenfist_utilities.api.error(), a third-party function pinned in pyproject.toml, so the body is built in another repository and phase 4 rewrites the returned response rather than changing that package. The stage is carried on the exception from scheduler.py's raise site rather than parsed out of the message. The 409 affinity branch is untouched. Update the OpenAPI declaration, within what its three-tuple response format can express -- it cannot declare a response header, which phase 4's D32 records as a filed gap rather than fixing here.

The contract this changes already has an owner: BaseTestCase.assertRefusedAtStage() (shakenfist/deploy/shakenfist_ci/base.py:1464), landed by the sizing plan's phase 3a and used by four assertions in cluster_ci_tests/test_saturation.py. Its docstring names this phase as the single place to change when the contract moves.

Client (client-python): carry response headers on APIException, and add an opt-in retry policy for 507 bodies carrying transient: true, bounded by the existing async-strategy deadline and reusing the shape of the 406 loop in _request_url. Off by default, and the CI suite leaves it off: the suite's clients use ASYNC_PAUSE, so a blind retry would spend up to 60 s inside each call the phase 2 wrapper makes, invisible to the wait records phase 5 reads as its denominator, and the saturation tests must never retry the refusal they are asserting. See phase 4's D33 for the full reasoning; an earlier draft of this section said the suite turns it on, and that was wrong.

Plan at medium effort; each half is small, and the coordination is a version pin between the two repositories.

Phase 5 -- Decide on server-side queued placement

A decision phase, not a build phase. Read the phase 2 wait summaries over at least twenty merge runs after the sizing plan's phase 4 has landed. If total wait per run is small and no test waits near its deadline, close this phase as Abandoned with the numbers and the reasoning in the phase file. Otherwise, design the queue against open question 8's constraints -- FIFO by request time, pinned creates admitted against their node only, IPAM allocated at placement rather than at request, a deadline the client's create ceiling can contain -- and record the reversal of scheduler-reservations D8 in that plan's decisions file before any code is written.

Plan at high effort if it goes ahead; the state-machine and fairness questions are the expensive kind.

Planned in PLAN-transient-capacity-refusals-phase-05-queue-decision.md, which fixes the numeric decision rule before the data is read, because "small" and "near its deadline" above are not numbers and a phase which reads twenty runs and then decides what those words meant is narrating a decision rather than taking one.

The gate has since opened -- the sizing plan's reshape landed as shakenfist/actions#88 (f78576e) and its step 4d reported -- and the first reading has been taken. It was unreadable: 92.3% of the window's instrumented bundles carry no capacity-wait trace, because the trace cannot express "nothing was refused" (#4337). The phase therefore extends its window once, remains In progress, and has decided nothing. The amendment that extension owed is now written as D43, before the second window was read. See open question 8 above and that plan's Outcome.

Phase 6 -- Documentation and close-out

Document the transient-refusal contract in docs/operator_guide/scheduler.md and the API reference, the suite wrapper and its allowlist marker in docs/developer_guide/ci.md, and the wait summary beside the headroom probe's documentation. Update docs/plans/index.md and the sibling plans' cross-references to their final state. Comment on #3772 with the before-and-after pass rate and close it only if the Debian 12 tier job's failures are no longer sufficient_idle_cpu; otherwise leave it open with the numbers.

Phase 7 -- Push audit

Runs PUSH-AUDIT.md over the accumulated diff of every phase in this plan, not the last phase's diff alone. Auditing one phase at a time would miss what the phases did to each other, which matters here more than it looks: phases 1, 2 and 3 are explicitly allowed to run in parallel, and phase 4's client retry is meant to carry the same semantics phase 2 proved, so a divergence between them is exactly the kind of defect no single phase can see.

By the time this runs its phases will have merged, and a diff against develop will be empty and would read as a clean audit. The baseline is therefore the Merged column in the Execution table above, not develop...HEAD. Findings land as their own pull request, and the plan is not complete until each is resolved or declined in writing here. If the audit finds nothing, that is recorded in one sentence.

Phase 2 may land partly outside this repository: its suite wrapper touches the CI harness. Where it does, its row names the repository, and that half is audited against that repository's default branch as part of the pull request that lands it, with this phase citing that audit rather than re-running it.

Phase 5 is a different situation needing a different response. It is a decision phase which may produce no code at all, and if it closes as Abandoned there is nothing here for the audit to read. That is recorded as such -- an audit which says what it had no diff to scope over is a result; one which reports a clean run over an empty range is not.

Agent guidance

Execution model

Sub-agent execution model (shared block; do not edit -- the canonical copy lives in shakenfist/development at templates/shared-blocks/subagent-execution-model.md):

All implementation work is done by sub-agents, never in the management session. The management session is reserved for planning, review, and decision-making. This keeps the management context lean and avoids drowning it in implementation diffs.

The workflow is:

1. Plan at high effort in the management session.
2. Spawn a sub-agent for each implementation step with the brief from the plan, at the recommended effort level and model.
3. Review the sub-agent's output in the management session. Check the actual files -- the sub-agent's summary describes what it intended, not necessarily what it did.
4. Fix or retry if the output is wrong. Diagnose whether the brief was insufficient (improve it) or the model was too light (upgrade it), then re-run.
5. Commit once the management session is satisfied.

This applies to all steps, including high-effort ones. If a sub-agent cannot succeed even with a detailed brief and the right model, that is a signal the brief needs improving, not that the management session should do the implementation itself.

Use isolation: "worktree" for sub-agents when the change is risky or experimental; the worktree is discarded if the output is unsatisfactory. For safe, well-understood changes, sub-agents can work directly in the main tree.

Planning effort

Planning effort (shared block; do not edit -- the canonical copy lives in shakenfist/development at templates/shared-blocks/plan-planning-effort.md):

The master plan itself is always created at high effort -- it requires broad codebase understanding, cross-referencing several source files, and judgment calls about scope and sequencing.

Each phase plan states the recommended effort level for planning that phase. Phases that turn on design decisions, cross-component coordination, protocol changes, or subtle correctness questions should be planned at high effort. Phases that are mechanical, or that follow a pattern already established elsewhere in the codebase, can be planned at medium effort.

In this project

Phases 1, 2 and 5 are planned at high effort: the first for its interaction with the stability gate and the degraded-read distinction, the second for its two-repository plumbing and the load-budget declaration, the fifth because it may be a state-machine design. Phases 3, 4 and 6 follow patterns that already exist and are planned at medium effort.

Step-level guidance

Sub-agent step guidance (shared block; do not edit -- the canonical copy lives in shakenfist/development at templates/shared-blocks/subagent-step-guidance.md):

Each phase plan includes a table like this:

Step	Effort	Model	Isolation	Brief for sub-agent
1a	medium	sonnet	none	One-sentence summary of what to do and which files to touch
1b	high	opus	worktree	Why this needs high effort: requires understanding X to do Y

Effort levels, from cheapest to most thorough:

• low -- Purely mechanical changes: rename, reformat, add a log line, regenerate generated code. The brief is a complete instruction.
• medium -- The plan provides enough context to follow a clear brief. The sub-agent may read a few files, but the approach is already decided.
• high -- Requires reading several files, making judgment calls, or understanding non-obvious invariants. The sub-agent needs to think about edge cases.
• xhigh -- The setting for hard coding and agentic steps: long-horizon changes, or steps where the sub-agent must both research and implement.
• max -- Correctness matters more than cost. Expect diminishing returns and occasional overthinking; reserve it for steps where a wrong answer would be expensive to detect.

Brief for sub-agent: this is the key field. Write it as if briefing a colleague who has never seen the codebase. Include what to change, which files to touch, what patterns to follow, and any non-obvious constraints.

A good brief front-loads the research the planner already did, so the implementing agent does not repeat it. Instead of "add storage functions for the new object", name the functions to add, the file they belong in, the existing equivalent to mirror (with line numbers), and any registration the change also needs.

The better the brief, the lower the effort level needed and the lighter the model that can succeed.

In this project

The invariants a brief in this plan must carry, because none of them is inferable from the code being edited:

• Placement transactions open with a guarded UPDATE, never a SELECT (the snapshot-isolation invariant, AGENTS.md). No phase here touches one, and a brief should say so explicitly so a sub-agent does not "improve" one in passing.
• Attribute writes carry a field mask (CLAUDE.md, common pitfall 3).
• Any new suite-side or daemon-side polling is declared in shakenfist/data/database_load_budget.yaml or HARNESS_DRIVEN_PAIRS, with prose naming the producer, or test_no_unbudgeted_fixed_rate_database_polling will fail the next merge group (#3975, #4028).
• The reconcile pass stays behind cluster_stable().
• The 409 affinity refusal is never retried.

Model choice

Sub-agent model roster (shared block; do not edit -- the canonical copy lives in shakenfist/development at templates/shared-blocks/subagent-model-roster.md):

The planner recommends which model is best suited to each step. This is a judgment call, not a rigid rule -- the right model depends on what the step requires, not on whether it is "planning" or "implementation". The models available to sub-agents are:

• fable -- The most capable model available, for the hardest reasoning and the longest-horizon work: multi-step changes a single sub-agent must carry end to end, or steps whose correctness depends on holding a whole subsystem in mind at once. It costs materially more than opus, so reserve it for steps that have already defeated opus or are expected to.
• opus -- The default for steps needing deep reasoning, architectural understanding, subtle correctness judgment (locking, state machines, migrations), or intricate implementation that would be costly to debug if it were wrong.
• sonnet -- A good default for well-briefed implementation work. Faster and cheaper than opus, and effective when the plan front-loads the research and the brief leaves no broad judgment calls to make.
• haiku -- Suitable for purely mechanical tasks: search-and-replace, regenerating generated code, adding log lines, running commands. The brief must be a near-complete instruction.

Model choice interacts with effort level and brief quality. A detailed brief compensates for a lighter model -- sonnet at medium effort with a thorough brief often matches opus at medium effort with a vague brief. The planner's job is to write briefs good enough that the recommended model can succeed.

The model also determines the context window: fable, opus and sonnet have 1M tokens, haiku has 200K. A step that must hold many files in context at once may need one of the larger-context models for that reason alone, even when the reasoning itself is straightforward.

When in doubt, skew to the more capable model. Saving money only matters if the outcome is still acceptable. A failed or low-quality implementation wastes more time -- and therefore more money -- than the heavier model would have cost. Recommend a lighter model only when you are confident the brief is detailed enough for it to succeed.

Management session review checklist

Management session review checklist (shared block; do not edit -- the canonical copy lives in shakenfist/development at templates/shared-blocks/plan-review-checklist.md):

After a sub-agent completes, the management session verifies:

• The files that were supposed to change actually changed -- read them, do not trust the summary.
• No unrelated files were modified.
• The changes match the intent of the brief: not merely syntactically correct, but semantically right.
• The project's own pre-merge checks pass, including any generated code that has to be regenerated and committed (see the project-specific checks below).
• The commit message follows project conventions, including the Co-Authored-By line recording model, context window, and effort level.

In this project

The project-specific checks referred to above are:

• The code passes pre-commit run --all-files (flake8, stestr unit tests, mypy).
• python3 tools/check-plan-status.py passes after any edit to a plan or to docs/plans/index.md.
• A change that adds polling has its budget entry, and test_no_unbudgeted_fixed_rate_database_polling has been reasoned about, not merely run.

Administration and logistics

Success criteria

We will know when this plan has been successfully implemented because the following statements will be true:

• In every merge-run bundle, every hypervisor has a capacity row before the run's first instance placed event, and no first reconcile pass records a drift_cpus above zero.
• A node's published cpu_total_instance_vcpus falls within ten seconds of its last instance being undefined.
• No functional test fails with 507 sufficient_idle_cpu at create_instance; a test that waits for capacity records how long, and the run's summary reports it.
• POST /instances refusals for capacity carry stage and a transient boolean, with Retry-After on the refusals a retry can actually clear, and shakenfist_client can be told to honour them.
• The decision on server-side queued placement is written down in the phase 5 file with the data it was made from, whichever way it went.
• The Debian 12 tier job's pass rate is comparable to the other cluster jobs, and its remaining failures are not sufficient_idle_cpu.
• The code passes pre-commit run --all-files (flake8, stestr unit tests, and mypy type checking).
• Lines are wrapped at 120 characters, single quotes for strings, double quotes for docstrings.
• Documentation in docs/ has been updated. ARCHITECTURE.md and AGENTS.md are updated only if a convention or the shape of the system changed; a transient-refusal contract is an operator-guide and API-reference matter.

Documentation index maintenance

This plan is registered in docs/plans/index.md (one row, in the Master plans table) and in docs/plans/order.yml. Phase files are linked from the Execution table above and appear in neither, which is what tools/check-plan-status.py enforces.

Plan close-out sections (shared block; do not edit -- the canonical copy lives in shakenfist/development at templates/shared-blocks/plan-closeout-sections.md):

Future work

We should list obvious extensions, known issues, unrelated bugs we encountered, and anything else we should one day do but have chosen to defer to here, so that we do not forget them.

• Re-derive the load budget once the change-triggered publish has run for a measurement window. Phase 3 marks the (operation, caller_daemon) pairs on the metrics publish path activity_coupled, which switches their enforcement off (shakenfist/deploy/shakenfist_ci/load_budget.py:443-450), because the publish rate is now coupled to instance churn and the existing fit was made against a fixed 60 s cadence. That is the honest declaration and it costs regression detection on the two of the four pairs which had it before -- GetQueueLength and GetNodeByFqdn; the other two are new entries which were unbudgeted rather than enforced. Running tools/derive-database-load-budget.py over a window that includes the new behaviour would give those pairs fitted activity-coupled rates again. It needs the change running on sfcbr for days, so it could not be part of phase 3. The marks survive re-derivation (tools/derive-database-load-budget.py:476), so this is an addition rather than a repair. Tracked as #4197. Found at phase 3's planning survey; see D23 there, and phase 3's Outcome for the pre-change figure the re-derivation has to be compared against.

• Make the metrics-drop test's rise attributable to its own instance. test_cluster_resources_measured_drops_after_delete (cluster_ci_tests/test_nodes.py) waits for a node's cpu_measured to rise before it takes the baseline it later asserts a drop against, but the rise predicate is a threshold on a shared node: in two of the three topologies read at phase 3's closeout it was already satisfied 0.34 s and 1.05 s after the domain started, which is sooner than any publish carrying that domain could arrive. The baseline can therefore include a vCPU the test did not create, and the drop can be satisfied by a sibling's instance going away. This is a false pass and not a false failure, so it does not destabilise CI. No node_metrics timestamp is published over REST, so the cheap fix is to ignore any rise read before a full 5 s domain-poll interval has elapsed since the create. Found at phase 3's closeout; see that phase's Outcome.

• Make get_all_domains() one libvirt call. Resolved: phase 1a of PLAN-power-state-correctness.md removed LibvirtConnection.get_all_domains(), which was listDomainsID() followed by a lookupByID() and a name() per domain, and moved its callers (daemons/cleaner/scheduled_tasks.py, daemons/resources/main.py) onto get_active_sf_domains(), the single-call listAllDomains(VIR_CONNECT_LIST_DOMAINS_ACTIVE) replacement. Found at phase 3's planning survey; see D22 there.

• Stop re-scheduling an already-charged placement. NodeInstNetdescOp._instance_preflight() (shakenfist/operations/node_inst_netdesc_op.py:156-162) constructs a fresh Scheduler -- a full refresh_metrics(), one get_node_metrics RPC per node -- and re-runs find_candidates(inst, candidates=[config.NODE_UUID]) for an instance the API already admitted on this node. The ledger term cannot refuse it (the self-charge is subtracted), but the 60-second-old measured_cpus can, and when it does the op redirects to another node and re-enqueues the artifact fetches. Replacing that with "is my placement still recorded here and is the node healthy" removes a distinct CI failure family (Too many start attempts) and halves per-create scheduler load. Not a 507 cause, so not in this plan.

• Suite concurrency denominated in ledger. The sizing plan's Future work already records this; phase 2's wait summary is the measurement that would justify it.
• The queue-depth stage. _has_reasonable_queue_state refuses any node with more than twenty waiting jobs. Under suite concurrency queue depth is exactly what spikes; it did not fire in the runs read here but it is untracked and worth a census line.
• An absent waits file reads as "unknown", not as zero. The phase 2 wrapper creates /srv/ci/traces/instance-waits.jsonl on its first write, so a run which was refused nothing leaves no file, and tools/ci_headroom_report.py --waits declines to call that zero waits -- correctly, since it is also what a component ref predating the wrapper and a run whose writes all failed look like. Phase 5 harvests over many bundles and cannot tell the three apart. Writing an empty file once at suite start-up would make "empty" mean zero and leave "absent" meaning the other two. Found at phase 2's closeout; see that phase's Outcome. Now fixed, after phase 5's first measurement window read 92.3% of its qualifying units as unknown (issue 4337): BaseTestCase.setUp() touches the file into existence via ensure_capacity_wait_trace(), which swallows every failure exactly as the append path does.
• Every refused create is a full create-and-delete. enqueue_delete_due_error at the 507 site means each refusal costs an object, IPAM allocations, an event trail and a delete op, at the moment the cluster is busiest. Phase 5's queue, if it is built, removes this; if it is not, the cost stands and should be measured. Phase 4's client review found the sharper version of it: because the client replays a marked request byte for byte, every replay pays that cost again, and an ASYNC_BLOCK caller with no timeout of its own had an hour of budget against a fixed fifteen second hint. The client now caps replays at TRANSIENT_RETRY_MAXIMUM_ATTEMPTS = 5 for that reason, which bounds the damage without removing it.
• The transient marker is a promise the server has not written anything. A client which replays a request byte for byte can only do so safely if the refusal committed no part of it. The create path is the marked endpoint today and it does commit -- see the bullet above -- so the promise currently rests on the delete that follows the refusal. The obligation matters most for the second endpoint anyone marks: send_upload()'s natural refusal is also a 507, and marking it transient without thinking about partial writes would be a data bug rather than a latency bug. Recorded in client-python's AGENTS.md and its retry documentation. Found at phase 4's client review.
• Split claim denials out of the transient marker. Phase 4 publishes capacity_guard as transient, which is honest only while mariadb.CLAIM_ENFORCEMENT_HARD is False and a namespace claim therefore cannot refuse a placement at all. When PLAN-scheduler-reservations phase 5 flips that constant, a claim denial becomes namespace quota exhaustion -- not something fifteen seconds fixes -- and it must stop inheriting capacity_guard's classification. The constraint is recorded at TRANSIENT_CAPACITY_STAGES in shakenfist/external_api/base.py, where the person making that change will be reading. Found at phase 4's review; see D35 there.
• Name the minimum client version, and retire two workarounds with it. APIException.headers is unreleased: the latest shakenfist-client release is v0.8.3 and no tag contains commit 80019a0. Until one does, assertRefusedAtStage() asserts the Retry-After header only when the exception exposes one, and docs/operator_guide/capacity_refusals.md warns operators that the retry_transient_capacity flag it documents is in no client they can install. Both are one edit each once a release exists.
• A transient marker for address-pool exhaustion. The two CongestedNetwork 507s are left bare by phase 4's D29. They probably do deserve a marker, with a horizon derived from the deletion halo rather than from defer()'s default, and the deletion halo is a much longer and differently-shaped wait. Needs its own evidence about how long the halo actually holds an address in practice, which is why phase 4 declined to guess.

Bugs fixed during this work

This section should list any bugs we encounter during development that we fixed. You should also scan the project's issue tracker, where one exists, for directly related issues that we should either resolve as part of this master plan or at least be aware of while planning it.

• #3772 (open, umbrella) -- the refusal this plan is about. Stays open until phase 6 has the before-and-after numbers.
• #4087 (open; #4106 attempted it and did not close it) -- the warm-up window. Phase 1 fixes it, comments with what #4106 did and did not cover, and closes it.
• #3498, #3602, #3670, #3728, #3749, #3767 (closed into #3772) -- the per-test victims. Do not file another; the umbrella exists because per-test tracking stopped paying for itself.
• #3907, #3565 (closed test-side) -- precedents for tolerating a transient refusal in the suite; phase 2 generalises what they did once.
• #3975, #4028 (closed) -- the headroom probe failing the load-budget check. The obligation they left is why phase 2 and phase 3 each carry a budget declaration.
• #1364 (open) -- lame-duck and evacuate. Where node-scoped reservations would earn their place; see open question 5.

Back brief

Before executing any step of this plan, please back brief the operator as to your understanding of the plan and how the work you intend to do aligns with that plan.

📝 Report an issue with this page