Development Plans

This section contains forward-looking roadmaps for Shaken Fist development. These documents describe planned features and architectural directions.

Forward-Looking Statements

Plans describe intended future work and may change based on implementation experience, community feedback, or shifting priorities. Check the status table below to see what has been implemented.

Plan sequencing

The set of incomplete plans has grown to the point where the order they land in matters. The intended sequencing is:

1. Network operations facade — complete. Landed via the network-facade branch.
2. Retire etcd — (absorbed into BYO MariaDB and sf-database tier phase 0; the standalone PLAN-remove-etcd.md has been removed.) Done. The data drain was already complete (the DATA_MIGRATIONS dict is empty), and the supporting machinery — shakenfist/etcd.py, etcd3gw, the etcd proto stubs, the drain test files, the migration-era sf-ctl aliases — was deleted as a single sweep by the BYO-MariaDB plan. The etcd_master ansible group rename landed with PLAN-remove-primary. What survives is deliberate: the vestigial is_etcd_master flag is pinned False for one release as a rollback fallback.
3. Health checks, readiness, and graceful drain — complete. A precondition for the BYO load-balancer story in remove-primary being operationally honest. Delivered sf-api /livez//readyz//healthz with SIGTERM drain, dependency-aware grpc.health.v1 on sf-database, systemd WATCHDOG liveness on the worker/elected daemons (which also closes the cluster-lock proof-of-life gap), and operator LB/upgrade docs. Landed on the health-checks branch. Node resource health is a sibling (complete) on a different axis — it drives node.state from the health of the storage/resource dependencies a node's hosted object types declare, so a dead disk or hung NFS mount takes a node out of scheduling. It was not sequenced against the BYO thread; it grew out of the sf-6 blob-NVMe incident and landed independently on the node-resource-health branch.
4. Remove the primary node — the BYO-infrastructure scope reduction. Complete: the deployer-level etcd_master → database_node rename landed with one-release fallbacks, and the deployer is now the shakenfist.shakenfist collection. Naturally followed by a wipe-and-redeploy of Mikal's production cluster against the new shape.
5. BYO MariaDB and sf-database as a tier — lifted out of remove-primary because it grew into its own master plan. Removes MariaDB-server install from the deployer entirely, reshapes sf-database into a deployer-chosen tier of equal stateless instances reached via client-side gRPC load balancing (not leader election), and carves schema/migration execution out of daemon startup into an operator-run sf-ctl ensure-mariadb-schema command. Complete. It landed in parallel with remove-primary's remaining phases, and its first phase performed the scope-shift edit to remove-primary itself.

Remove syslog forwarding (ship logs to Loki) delivers the "Loki-shipper story" that remove-primary phase 1 is explicitly gated on — it adds structured-JSON logging and an in-process, on-disk-spooled Loki push (modelled on the eventlog spool/drainer) before deleting the rsyslog wiring. It can land in parallel with the other BYO work and is sequenced ahead of remove-primary phase 1, which it realises.

The remaining incomplete plans — Embrace TLS, Sticky blob transfers, Replace exec'd network commands with netlink, Atomic scheduling via reservations, the connected Generic allocator / Network service ports / Network carrier model triple, and the not-yet-drafted OpenTelemetry instrumentation thread — are intentionally not ordered relative to each other here. They each have specific dependencies on either remove-primary having established the BYO shape (the operator-provides-PKI surface for TLS, the streaming-proxy baseline for sticky transfers, the sf-database election pattern for the others) or network-facade having landed (the netlink plan, whose privilege-separation phases need network-facade's single-mutator property), but among themselves the order is a triage decision still to be made now that remove-primary has landed. The scheduler-reservations plan is independent of the BYO shape but benefits from the OpenTelemetry thread landing first so that phase 0's design choices can be informed by real load and contention numbers.

Database load reduction sits outside that triage: it addresses a measured production problem (the sf-database tier serving ~527 ops/second at idle, 57% of it one polling loop). Phases 1–5 are complete and took that to 89–92 ops/second at its best, largely by removing fixed-rate idle polling and restoring the etcd-era “objects are cacheable, attributes of objects are not” principle in a MariaDB form. Its phase 4 (caller attribution via gRPC metadata and a per-caller counter) is also the first concrete slice of the OpenTelemetry instrumentation thread — the caller-identity plumbing is what a later span-propagation phase would reuse, it is designed to compose with the mTLS peer-identity model from Embrace TLS rather than duplicate it, and it is what made every subsequent diagnosis possible. The gain has since partially rotted — load is back to ~142 ops/second — so phase 6 chases the residual load and that regression, and phase 7 makes regression detection something any deployer can run rather than something only our own operations tooling can see.

The generic-allocator / network-service-ports / network-carrier-model triple is internally ordered. Generic-allocator is the foundational refactor (replaces five ad-hoc allocators with one primitive and is independently shippable). Network-service-ports builds on the allocator to expose per-network DNAT'd ports for managed services (web consoles, transfer agents, managed VPN endpoints). Network-carrier-model layers a smeared lease-based per-network carrier role with VIP advertisement on top, removing the network-node singleton; it depends on both prior plans and is the largest of the three. The triple supersedes the "network node failover" thread that was previously a not-yet-drafted line item.

Declarative API input validation is independent of all of the above and gated on nothing, so it can start whenever there is appetite. It closes a cluster of reported issues going back to 2020 (#528 and #936 are the parent issues) whose common cause is that request body values reach handlers untyped. It is drafted now because a per-endpoint fix for one instance of it (#3609) was attempted and abandoned: seven hand-rolled guards, two of them wrong on the first attempt, and the defect class still not covered. Its one coordination point is with the API query batching roadmap, which needs the same bounded limit/offset parameter types for #1974.

Agent operation deadlines and progress detection is similarly independent and gated on nothing. It is driven by CI reliability, but the defect is user-facing too: the #3516 sf-sidechannel wedge (a get-file stuck in executing until the client times out, with the agent-side trigger tracked as #2240) is one of the two most frequent merge-queue flakes as of 2026-08, and because each instance runs at most one executor, a wedged operation also blocks every other agent operation against that instance until the 900-second backstop fires. It adds client-propagated wall-clock deadlines and per-command progress timeouts to agent operations, plus a retry path for operations that fail in executing. Its API-facing phase follows the parameter declaration rules that Declarative API input validation enforces, but does not depend on that plan's remaining phases. Its follow-on, Dependency-aware agent operations, extends the cluster operation depends_on/runs_after vocabulary to agent operations (including cross-instance edges, enabling fire-and-await fleet orchestration such as rolling updates) and elevates documentation-with-worked-examples to a deliverable, with client-python-k3s as the first of an example-application suite; it is explicitly gated on the deadlines plan landing, because dependency waiting relies on deadline expiry to break user-created cycles.

Bound the size of DatabaseService gRPC replies is independent and gated on nothing, but it is driven by a production defect rather than an ambition, so it should not sit indefinitely. Several DatabaseService replies are unbounded by construction and the only ceiling on them is gRPC's message size limit, which the sfcbr cluster has now crossed twice from two daemons on two RPCs (#3638). A stopgap has raised the client cap and made the affected callers honest about failed reads — including one that was using an unreadable list as a licence to delete a node's blob store — but the protocol still permits replies whose size we discover by having one fail. Its phase 2 (sweep callers ask for a bounded page rather than everything) is small and independently shippable; its phase 3 is a protocol change and inherits the eventlog plan's deferred cursor-pagination work.

The blob-storage and SQL-pushdown roadmaps and the network-facade plan run on their own cadence and are not part of this sequencing. Neither are the plans which appear only in the table below: sf-netserv, CI node-exec assertions, attribute field masks, retry transient artifact fetches, fix cluster_operation_targets uniqueness and Kerbside VDI console tokens — the first is a thoughtbubble and the rest are self-contained pieces of work that were never sequenced against this thread.

Status vocabulary

The Status column below holds exactly one of these terms and nothing else. It is the whole-plan status, so it only reads Complete once every phase has landed. Where a plan is In progress or Blocked for a reason the Phases column does not make obvious, that reason belongs in the plan itself rather than in this table. The canonical wording is the plan-status-vocabulary shared block in PLAN-TEMPLATE.md.

Status	Meaning
Proposed	Written down as a concept, not yet scheduled.
Not started	Scheduled, but no work has begun.
In progress	Work has begun and has not finished.
Blocked	Cannot proceed until something outside the plan changes.
Complete	The work is done.
Abandoned	Deliberately dropped without being done.
Superseded	Replaced by another plan, which the plan names.

Master plans

Most master plans track their own phases -- a phase plan and a status for each -- in their own Execution table, and the Phases column here is arithmetic over that table; follow the plan link for what each phase covers, and for why a plan is where it is. It counts completed phases, so a plan whose phases were abandoned or superseded rather than done can read Complete without the two numbers meeting, and — means the plan has no enumerated phases yet, or has a table which is still a placeholder rather than a phase list.

Four plans are counted by hand because they do not keep a phase table: blob storage, API query batching and attribute field masks carry their phases as headings, and queue performance is numbered in steps rather than phases. Two more publish no arithmetic because their tables are placeholders: owning more of the QEMU stack names three phases and then an ellipsis, and artifact UX rework a decisions pass and "(later phases)". These six are the checker's blind spot, and the only numbers here it cannot recompute.

Date	Plan	Intent	Status	Phases
2026-01-08	Blob storage roadmap	Composite blobs, lazy deduplication and content-defined chunking	In progress	1 of 3
2026-01-10	API query batching	Batch and prefetch related objects so list endpoints stop issuing per-object queries	Not started	0 of 3
2026-04-23	SQL-pushdown filtering	Push object filtering into MariaDB instead of scanning and filtering in Python	Complete	7 of 7
2026-05-06	Replace last_cluster_operation	Trade the single-pointer gate for an append-only cluster_operation_targets history	Complete	5 of 5
2026-05-14	Fix cluster_operation_targets uniqueness	Composite UNIQUE so a multi-target operation records all of its target rows	Complete	—
2026-05-15	Network operations facade	Split Network into a queue-enqueuing facade and a single-mutator worker	Complete	10 of 10
2026-05-16	Embrace TLS	Operator-provided PKI and TLS on every internal and external listener	Not started	0 of 8
2026-05-16	Recurring cluster operations	A cron-like framework absorbing the scheduled-task loops, plus user-facing recurrence	Proposed	0 of 7
2026-05-16	Remove the primary node	Retire the special primary node in favour of a BYO-infrastructure deployer	Complete	5 of 7
2026-05-16	Sticky blob transfers	Session affinity for blob transfers, deferred until OpenTelemetry supplies upload-path numbers	Not started	0 of 5
2026-05-17	Retry transient artifact fetches	A per-operation retry budget, so a network blip during a fetch no longer errors the artifact	Complete	—
2026-05-17	Health checks, readiness and drain	/livez, /readyz, /healthz, gRPC health, watchdog liveness and SIGTERM drain	Complete	5 of 5
2026-05-20	Replace exec'd network commands with netlink	Native netlink calls in place of shelling out to ip, bridge and friends	Not started	0 of 7
2026-05-22	Eventlog direct to MariaDB	Remove the eventlog service and write events straight to MariaDB	Complete	7 of 7
2026-05-22	Generic allocator	Replace five ad-hoc finite-resource allocators with a single primitive	Not started	0 of 7
2026-05-22	Network carrier model	A lease-based per-network carrier with VIP advertisement, retiring the network-node singleton	Not started	0 of 12
2026-05-22	Network service ports	Per-network DNAT'd ports for managed services	Not started	0 of 7
2026-05-22	Atomic scheduling via reservations	Guarded-UPDATE capacity counters and namespace claims in place of read-then-place scheduling	In progress	4 of 10
2026-05-24	Queue performance and coalescing	Batched dequeue, and worker- and enqueue-side dedup of redundant cluster operations	In progress	6 of 7
2026-06-01	OIDC authentication	Federated login against an external OIDC provider	Not started	0 of 10
2026-06-02	Owning more of the QEMU stack	Direct QMP control, and perhaps one day libvirt's job as well	Not started	—
2026-06-03	BYO MariaDB and sf-database as a tier	A deployer-chosen database tier reached by client-side gRPC load balancing	Complete	8 of 8
2026-06-12	Remove the Apache load balancer	An operator-provided load balancer in place of the bundled Apache	Complete	2 of 2
2026-06-13	Artifact UX rework	Rework the artifact, blob, label, upload and snapshot user interface	Proposed	—
2026-06-19	Remove syslog forwarding	Structured JSON logging shipped to Loki, replacing the rsyslog wiring	Complete	7 of 7
2026-07-13	CI node-exec assertions	Run an assertion on a named cluster node, and the floating-IP and network lifecycle tests that use it	Complete	—
2026-07-14	Workload identity federation	Federated workload identity and first-class namespace keys	Complete	7 of 7
2026-07-17	Attribute field masks everywhere	Masked attribute writes, and the node instances list un-packed into object_references	Complete	2 of 2
2026-07-19	Truthful cluster operation visibility	An observational flag, so "is anything in flight?" stops counting background housekeeping	In progress	2 of 6
2026-07-19	Database load reduction	Cut steady-state MariaDB load from the sf-database tier, and keep it cut	In progress	5 of 7
2026-07-19	Kerbside VDI console tokens	Cluster-signed tokens exchanged for a VDI console session, across four repositories	In progress	9 of 10
2026-07-19	Node resource health	Declarative dependency checks driving node state, so a dead disk stops scheduling	Complete	5 of 5
2026-07-21	Per-host resource reservations	Per-node RAM, CPU and disk reservation overrides	Complete	4 of 4
2026-07-28	sf-netserv	Replace dnsmasq with a Rust per-network service plane	Proposed	—
2026-08-03	API input validation	Declarative request validation and a consistent error contract for the REST API	In progress	3 of 7
2026-08-14	Agent operation deadlines	Client-propagated deadlines and per-command progress timeouts for agent operations	In progress	2 of 8
2026-08-14	Dependency-aware agent operations	depends_on and runs_after for agent operations, including cross-instance edges	Blocked	—
2026-08-16	Bound gRPC reply sizes	Make DatabaseService replies bounded by construction rather than by the message size limit	Not started	0 of 6

📝 Report an issue with this page