Development Plans

This section contains forward-looking roadmaps for Shaken Fist development. These documents describe planned features and architectural directions.

Forward-Looking Statements

Plans describe intended future work and may change based on implementation experience, community feedback, or shifting priorities. Check the status table below to see what has been implemented.

Plan sequencing

The set of incomplete plans has grown to the point where the order they land in matters. The intended sequencing is:

1. Network operations facade — complete. Landed via the network-facade branch.
2. Retire etcd — (absorbed into BYO MariaDB and sf-database tier phase 0; the standalone PLAN-remove-etcd.md has been removed.) Done. The data drain was already complete (the DATA_MIGRATIONS dict is empty), and the supporting machinery — shakenfist/etcd.py, etcd3gw, the etcd proto stubs, the drain test files, the migration-era sf-ctl aliases — was deleted as a single sweep by the BYO-MariaDB plan. The etcd_master ansible group rename landed with PLAN-remove-primary. What survives is deliberate: the vestigial is_etcd_master flag is pinned False for one release as a rollback fallback.
3. Health checks, readiness, and graceful drain — complete. A precondition for the BYO load-balancer story in remove-primary being operationally honest. Delivered sf-api /livez//readyz//healthz with SIGTERM drain, dependency-aware grpc.health.v1 on sf-database, systemd WATCHDOG liveness on the worker/elected daemons (which also closes the cluster-lock proof-of-life gap), and operator LB/upgrade docs. Landed on the health-checks branch. Node resource health is a sibling (complete) on a different axis — it drives node.state from the health of the storage/resource dependencies a node's hosted object types declare, so a dead disk or hung NFS mount takes a node out of scheduling. It was not sequenced against the BYO thread; it grew out of the sf-6 blob-NVMe incident and landed independently on the node-resource-health branch.
4. Remove the primary node — the BYO-infrastructure scope reduction. Complete: the deployer-level etcd_master → database_node rename landed with one-release fallbacks, and the deployer is now the shakenfist.shakenfist collection. Naturally followed by a wipe-and-redeploy of Mikal's production cluster against the new shape.
5. BYO MariaDB and sf-database as a tier — lifted out of remove-primary because it grew into its own master plan. Removes MariaDB-server install from the deployer entirely, reshapes sf-database into a deployer-chosen tier of equal stateless instances reached via client-side gRPC load balancing (not leader election), and carves schema/migration execution out of daemon startup into an operator-run sf-ctl ensure-mariadb-schema command. Complete. It landed in parallel with remove-primary's remaining phases, and its first phase performed the scope-shift edit to remove-primary itself.

Remove syslog forwarding (ship logs to Loki) delivers the "Loki-shipper story" that remove-primary phase 1 is explicitly gated on — it adds structured-JSON logging and an in-process, on-disk-spooled Loki push (modelled on the eventlog spool/drainer) before deleting the rsyslog wiring. It can land in parallel with the other BYO work and is sequenced ahead of remove-primary phase 1, which it realises.

The remaining incomplete plans — Embrace TLS, Sticky blob transfers, Replace exec'd network commands with netlink, Atomic scheduling via reservations, the connected Generic allocator / Network service ports / Network carrier model triple, and the not-yet-drafted OpenTelemetry instrumentation thread — are intentionally not ordered relative to each other here. They each have specific dependencies on either remove-primary having established the BYO shape (the operator-provides-PKI surface for TLS, the streaming-proxy baseline for sticky transfers, the sf-database election pattern for the others) or network-facade having landed (the netlink plan, whose privilege-separation phases need network-facade's single-mutator property), but among themselves the order is a triage decision still to be made now that remove-primary has landed. The scheduler-reservations plan is independent of the BYO shape but benefits from the OpenTelemetry thread landing first so that phase 0's design choices can be informed by real load and contention numbers.

Database load reduction sits outside that triage: it addresses a measured production problem (the sf-database tier serving ~527 ops/second at idle, 57% of it one polling loop). All eight phases are complete. Phases 1–6 took that to ~102 ops/second on a quiet cluster, largely by removing fixed-rate idle polling and restoring the etcd-era “objects are cacheable, attributes of objects are not” principle in a MariaDB form. Its phase 4 (caller attribution via gRPC metadata and a per-caller counter) is also the first concrete slice of the OpenTelemetry instrumentation thread — the caller-identity plumbing is what a later span-propagation phase would reuse, it is designed to compose with the mTLS peer-identity model from Embrace TLS rather than duplicate it, and it is what made every subsequent diagnosis possible. Phase 6 chased what looked like a rot back to ~142 ops/second and found that most of it was the counter learning to see two more nodes and the cluster daemon (#3708) rather than new load — so the earlier “target met” reading is withdrawn, though the hunt still turned up two real defects worth ~21 ops/second (#3814, #3655). Its durable output is a load model with a per-node term, and phase 7 made regression detection against that model something any deployer can run rather than something only our own operations tooling can see. Phase 8 then audited the whole plan as one body of work under PUSH-AUDIT.md, which nine separate pull request reviews never did: the blocking findings were fixed on the way through, the remainder filed as #3942, #3943 and #3944, and the automated review of the audit’s own fixes found two further defects — one of them predicted by a test-coverage gap the audit had written down and not acted on.

The generic-allocator / network-service-ports / network-carrier-model triple is internally ordered. Generic-allocator is the foundational refactor (replaces five ad-hoc allocators with one primitive and is independently shippable). Network-service-ports builds on the allocator to expose per-network DNAT'd ports for managed services (web consoles, transfer agents, managed VPN endpoints). Network-carrier-model layers a smeared lease-based per-network carrier role with VIP advertisement on top, removing the network-node singleton; it depends on both prior plans and is the largest of the three. The triple supersedes the "network node failover" thread that was previously a not-yet-drafted line item.

Declarative API input validation is independent of all of the above and gated on nothing, so it can start whenever there is appetite. It closes a cluster of reported issues going back to 2020 (#528 and #936 are the parent issues) whose common cause is that request body values reach handlers untyped. It is drafted now because a per-endpoint fix for one instance of it (#3609) was attempted and abandoned: seven hand-rolled guards, two of them wrong on the first attempt, and the defect class still not covered. Its one coordination point is with the API query batching roadmap, which needs the same bounded limit/offset parameter types for #1974.

Agent operation deadlines and progress detection is similarly independent and gated on nothing. It is driven by CI reliability, but the defect is user-facing too: the #3516 sf-sidechannel wedge (a get-file stuck in executing until the client times out, with the agent-side trigger tracked as #2240) is one of the two most frequent merge-queue flakes as of 2026-08, and because each instance runs at most one executor, a wedged operation also blocks every other agent operation against that instance until the 900-second backstop fires. It adds client-propagated wall-clock deadlines and per-command progress timeouts to agent operations, plus a retry path for operations that fail in executing. Its API-facing phase follows the parameter declaration rules that Declarative API input validation enforces, but does not depend on that plan's remaining phases. Its follow-on, Dependency-aware agent operations, extends the cluster operation depends_on/runs_after vocabulary to agent operations (including cross-instance edges, enabling fire-and-await fleet orchestration such as rolling updates) and elevates documentation-with-worked-examples to a deliverable, with client-python-k3s as the first of an example-application suite; it is explicitly gated on the deadlines plan landing, because dependency waiting relies on deadline expiry to break user-created cycles.

Bound the size of DatabaseService gRPC replies is independent and gated on nothing, but it is driven by a production defect rather than an ambition, so it should not sit indefinitely. Several DatabaseService replies are unbounded by construction and the only ceiling on them is gRPC's message size limit, which the sfcbr cluster has now crossed twice from two daemons on two RPCs (#3638). A stopgap has raised the client cap and made the affected callers honest about failed reads — including one that was using an unreadable list as a licence to delete a node's blob store — but the protocol still permits replies whose size we discover by having one fail. Its phase 2 (sweep callers ask for a bounded page rather than everything) is small and independently shippable; its phase 3 is a protocol change and inherits the eventlog plan's deferred cursor-pagination work.

Right-size the CI test clouds is independent of all of the above and gated on nothing. It is driven by measurement rather than ambition: the five-hypervisor test cloud each CI job builds was an arbitrary choice that had never been checked, and checking it found that what binds is the scheduler's admission ledger (a 4 vCPU node yields 6 admitted vCPU, a 4 vCPU network node yields 3) rather than real CPU or memory, which sit at roughly 18% and 50% of allocation. The three-node slim-tier topology is below that line and passes 19% of merge runs, which is where the #3772 507 family comes from. The plan instruments headroom first, converts the coverage we currently get from scarcity by accident into explicit tests, and only then reshapes the topologies -- deliberately, because a bigger cloud would otherwise close #3772 and #3907 by silencing them. #3565 left that corpus when scheduler-reservations phase 6 closed it on a test change: test_affinity now skips rather than passes when the candidate set collapsed before affinity was scored, so a bigger cloud can no longer turn it green by hiding the question.

A capacity refusal is transient is the sibling that plan and scheduler-reservations both deferred to: what the system does about "no capacity right now" once the ledger is correct and the cloud is the size it should be. It was written on 2026-09-08 from a journal reading of six post-#4106 failing merge runs, which found that every sufficient_idle_cpu refusal was a pinned create onto a 4-thread infra hypervisor whose ledger of 3 was genuinely full while the cluster was not, that #4106's one-shot reconcile fires before any hypervisor has published and so leaves the warm-up window open in CI, and that a node keeps refusing for a metrics period after its instances are deleted. It closes the window for real, makes the functional suite wait for the node it needs and report how long it waited, publishes node metrics when the running-domain set changes, gives the API a Retry-After and a machine-readable transient marker, and defers the question of a server-side placement queue -- a reversal of scheduler-reservations D8 -- to a decision phase fed by the suite's own wait numbers. It owns nothing the sizing plan owns: topology is that plan's phase 4, and the evidence here is written into it.

The blob-storage and SQL-pushdown roadmaps and the network-facade plan run on their own cadence and are not part of this sequencing. Neither are the plans which appear only in the table below: sf-netserv, CI node-exec assertions, attribute field masks, retry transient artifact fetches, fix cluster_operation_targets uniqueness and Kerbside VDI console tokens — the first is a thoughtbubble and the rest are self-contained pieces of work that were never sequenced against this thread.

Status vocabulary

The Status column below holds exactly one of these terms and nothing else. It is the whole-plan status, so it only reads Complete once every phase has landed. Where a plan is In progress or Blocked for a reason the Phases column does not make obvious, that reason belongs in the plan itself rather than in this table. The canonical wording is the plan-status-vocabulary shared block in PLAN-TEMPLATE.md.

Status	Meaning
Proposed	Written down as a concept, not yet scheduled.
Not started	Scheduled, but no work has begun.
In progress	Work has begun and has not finished.
Blocked	Cannot proceed until something outside the plan changes.
Complete	The work is done.
Abandoned	Deliberately dropped without being done.
Superseded	Replaced by another plan, which the plan names.

Master plans

Most master plans track their own phases -- a phase plan and a status for each -- in their own Execution table, and the Phases column here is arithmetic over that table; follow the plan link for what each phase covers, and for why a plan is where it is. It counts completed phases, so a plan whose phases were abandoned or superseded rather than done can read Complete without the two numbers meeting, and — means the plan has no enumerated phases yet, or has a table which is still a placeholder rather than a phase list.

Three plans are counted by hand because they do not keep a phase table: blob storage, API query batching and attribute field masks carry their phases as headings. Two more publish no arithmetic because their tables are placeholders: owning more of the QEMU stack names three phases and then an ellipsis, and artifact UX rework a decisions pass and "(later phases)". These five are the checker's blind spot, and the only numbers here it cannot recompute.

Date	Plan	Intent	Status	Phases
2026-01-08	Blob storage roadmap	Composite blobs, lazy deduplication and content-defined chunking	In progress	1 of 4
2026-01-10	API query batching	Batch and prefetch related objects so list endpoints stop issuing per-object queries	Not started	0 of 4
2026-04-23	SQL-pushdown filtering	Push object filtering into MariaDB instead of scanning and filtering in Python	Complete	7 of 7
2026-05-06	Replace last_cluster_operation	Trade the single-pointer gate for an append-only cluster_operation_targets history	Complete	5 of 5
2026-05-14	Fix cluster_operation_targets uniqueness	Composite UNIQUE so a multi-target operation records all of its target rows	Complete	—
2026-05-15	Network operations facade	Split Network into a queue-enqueuing facade and a single-mutator worker	Complete	10 of 10
2026-05-16	Embrace TLS	Operator-provided PKI and TLS on every internal and external listener	Not started	0 of 9
2026-05-16	Recurring cluster operations	A cron-like framework absorbing the scheduled-task loops, plus user-facing recurrence	Proposed	0 of 8
2026-05-16	Remove the primary node	Retire the special primary node in favour of a BYO-infrastructure deployer	Complete	5 of 7
2026-05-16	Sticky blob transfers	Session affinity for blob transfers, deferred until OpenTelemetry supplies upload-path numbers	Not started	0 of 6
2026-05-17	Retry transient artifact fetches	A per-operation retry budget, so a network blip during a fetch no longer errors the artifact	Complete	—
2026-05-17	Health checks, readiness and drain	/livez, /readyz, /healthz, gRPC health, watchdog liveness and SIGTERM drain	Complete	5 of 5
2026-05-20	Replace exec'd network commands with netlink	Native netlink calls in place of shelling out to ip, bridge and friends	Not started	0 of 8
2026-05-22	Eventlog direct to MariaDB	Remove the eventlog service and write events straight to MariaDB	Complete	7 of 7
2026-05-22	Generic allocator	Replace five ad-hoc finite-resource allocators with a single primitive	Not started	0 of 8
2026-05-22	Network carrier model	A lease-based per-network carrier with VIP advertisement, retiring the network-node singleton	Not started	0 of 13
2026-05-22	Network service ports	Per-network DNAT'd ports for managed services	Not started	0 of 8
2026-05-22	Atomic scheduling via reservations	Guarded-UPDATE capacity counters and namespace claims in place of read-then-place scheduling	In progress	11 of 14
2026-05-24	Queue performance and coalescing	Batched dequeue and cluster-operation coalescing. The wait tail is gone and explicit fairness was not needed, though those numbers were measured while coalescing was inert -- a push audit found it had never worked (#3878), and review of that fix found it would have folded per-node mesh ops across nodes (#3884). Reopened for three phases: proving coalescing works on a running cluster and measuring what it costs, the flat 15 second dependency wait (#3863), and a multi-column fold key (#3884). Phase 9 has now measured sfcbr for 42 hours: the fold costs a 3.7 ms median rather than the ~200 ms the code asserted, and it matched 7 times in 1,335 attempts -- so coalescing is confirmed working and confirmed nearly inert on this workload, and the functional test is known to fail when coalescing is disabled. Phase 10 decomposed wait_seconds and withdrew phase 9's unexplained 15-17 second population: that was pre-#3916 traffic in a window whose published label was ten hours out, and after the back-off fix there is no such population at all. The deep tail that remains is queue sit and not deferral -- over 90% of the operations waiting 15 seconds or more never deferred once, mostly blob transfers on a lane saturated with their own work. One networknode/background incident, 12 operations drained serially 31 minutes late while the same dispatcher served that lane throughout, cannot be diagnosed from the events we retain, and carries forward as #3974, which asks for the two timestamps that would separate database queue sit from time inside the daemon. Phase 11 has built the multi-column coalescing key (#3884), scoped to network_ensure_mesh on sf-net only, because network_apply_create_hypervisor runs on sf-queues, whose worker pool has no per-target routing and so cannot support the fold safely -- that half is deferred to #4017. Both coalescing guards became key-aware and family-aware rather than merely key-aware, after two mid-phase corrections: a None key value has to bind IS NULL or widening the key would have silently disabled the only coalescing the cluster already does, and a key naming node_uuid turned out to be necessary but not sufficient, since the queue family decides which dispatcher drains the work. Verified against a real database: a node A survivor folds only node A's sibling, and narrowing the key back to the network alone reproduces the phase 8 cross-node bug exactly. Step 11h has now measured two equal 19 hour sfcbr windows either side of the deployment and the result is a clean negative: the guard the phase existed to remove went from 2,243 refusals to zero and the fold now runs 1,524 times on per-node network queues, but those folds collapsed no siblings at all, and the enqueue-side dedup did not absorb the work either (210 reuses before, 196 after). Survey finding 1's 62% ceiling was a far looser bound than it read as -- arriving in one dispatcher batch is not sharing a coalescing key, because ensure_mesh fans out per node for a single network and a node's queue fills with several different networks. The mechanism is not broken, which was checked separately: the same path folded four on the cluster-wide lane and the functional test asserts a per-node fold on every merge. Decision 6 stands -- the fold's median cost did not move (3.6ms to 4.1ms, uniform at 3.8-4.5ms across all 13 queues issuing folds), and the tail that did move belongs to the single busiest node rather than to the wider key, so a composite index has no scan cost to reduce. No deadlocks and no lock wait timeouts in either window	Complete	11 of 11
2026-06-01	OIDC authentication	Federated login against an external OIDC provider	Not started	0 of 11
2026-06-02	Owning more of the QEMU stack	Direct QMP control, and perhaps one day libvirt's job as well	Not started	—
2026-06-03	BYO MariaDB and sf-database as a tier	A deployer-chosen database tier reached by client-side gRPC load balancing	Complete	8 of 8
2026-06-12	Remove the Apache load balancer	An operator-provided load balancer in place of the bundled Apache	Complete	2 of 2
2026-06-13	Artifact UX rework	Rework the artifact, blob, label, upload and snapshot user interface	Proposed	—
2026-06-19	Remove syslog forwarding	Structured JSON logging shipped to Loki, replacing the rsyslog wiring	Complete	7 of 7
2026-07-13	CI node-exec assertions	Run an assertion on a named cluster node, and the floating-IP and network lifecycle tests that use it	Complete	—
2026-07-14	Workload identity federation	Federated workload identity and first-class namespace keys	Complete	7 of 7
2026-07-17	Attribute field masks everywhere	Masked attribute writes, and the node instances list un-packed into object_references	Complete	2 of 2
2026-07-19	Truthful cluster operation visibility	An observational flag, so "is anything in flight?" stops counting background housekeeping	In progress	2 of 7
2026-07-19	Database load reduction	Cut steady-state MariaDB load from the sf-database tier, and keep it cut	Complete	8 of 8
2026-07-19	Kerbside VDI console tokens	Cluster-signed tokens exchanged for a VDI console session, across four repositories. The cross-repository push audit closed the plan, and it was not a formality: it found a blocking, destructive defect that eight pull-request reviews across four repositories had not. A failed signing-key fetch errored the Shaken Fist console source, an errored source is skipped before the scrape loop which keeps its consoles alive, and the unconditional cleanup afterwards then deleted that source's entire console inventory -- breaking the direct and proxy console routes, which have nothing to do with tokens. It is the third defect from the one configuration no phase ever tested (Kerbside integration not configured, after #4003 and #4009) and the first that destroys data. Fixing it took two pull requests: kerbside#412 removed the trigger without touching main.py, so every other early-exit path -- an unknown source type, an exception mid-generator, the CA fetch which sits outside any try -- still deleted the inventory, and kerbside#413 narrowed the cleanup to the sources a pass actually enumerated to exhaustion. Both ship in Kerbside v0.6.0; earlier releases need the signing key provisioned before the cluster is upgraded, which the operator guide and release notes now say. The audit also found one high-severity finding in ryll, whose SPICE TLS verifier trusted the public WebPKI root set even when a .vv supplied a private cluster CA -- the mechanism this plan's whole host_subject story depends on (ryll#358, merged). Three medium security findings and the audit's own discovery that the .vv type collapse had unit but no functional coverage were fixed in the audit's own pull request; fourteen advisory findings were filed as issues across the four repositories	Complete	12 of 12
2026-07-19	Node resource health	Declarative dependency checks driving node state, so a dead disk stops scheduling	Complete	5 of 5
2026-07-21	Per-host resource reservations	Per-node RAM, CPU and disk reservation overrides	Complete	4 of 4
2026-07-28	sf-netserv	Replace dnsmasq with a Rust per-network service plane	Proposed	—
2026-08-03	API input validation	Declarative request validation and a consistent error contract for the REST API	In progress	4 of 8
2026-08-14	Agent operation deadlines	Client-propagated deadlines and per-command progress timeouts for agent operations; audited, and the one defect of its own making the audit found (#4074) was fixed hours later by #4080 -- see Known defects in the plan for the two which remain open	Complete	9 of 9
2026-08-14	Dependency-aware agent operations	depends_on and runs_after for agent operations, including cross-instance edges	Blocked	—
2026-08-16	Bound gRPC reply sizes	Make DatabaseService replies bounded by construction rather than by the message size limit	Not started	0 of 7
2026-08-27	Right-size the CI test clouds	Measured sizing for the nested CI clouds, with headroom instrumentation and explicit saturation coverage before any cloud grows. The instrument now runs in every cluster job, and its first readings show the cluster-wide utilisation figure the plan was written against can be half the per-node figure the scheduler actually admits against. Phase 3 turns that reading into a design constraint: its survey found the suite has no serialisation seam, so the saturation test the plan asked for fills one hypervisor rather than the cluster, and proves the refusal contract with a request no cluster could satisfy	In progress	3 of 7
2026-09-08	A capacity refusal is transient	Close the capacity-ledger warm-up window for real, make the functional suite wait for the node it needs and report the wait, publish metrics on domain-set change, and give capacity refusals a Retry-After; the #3772 family's transient-condition half. Phase 1 measured its own effect on slim-tier: the same probe read 165 s of unguarded placement before it and 0 s after, and closed #4087, which #4106 had left open and #4144 had closed on the mistaken premise that #4106 fixed it	In progress	1 of 6

📝 Report an issue with this page