Skip to content

Database internals

How Shaken Fist uses MariaDB beyond the deployment and schema concerns covered in the operator guide: the static object value cache, filter-pushdown discipline, gRPC reliability, and the cluster operation tracking and work-queue machinery.

+----------------+     +----------------+     +----------------+
|   sf-api       |     |   sf-cleaner   |     |   sf-queues    |
|   sf-net       |     |   sf-cluster   |     |   sf-resources |
+-------+--------+     +-------+--------+     +-------+--------+
        |                      |                      |
        +----------------------+----------------------+
                               |
                         gRPC (13005)
                               |
                    +----------+----------+
                    |    sf-database      |
                    +----------+----------+
                               |
                        +------+------+
                        |   MariaDB   |
                        | (state,     |
                        |  IPAM,      |
                        |  uploads,   |
                        |  blobs,     |
                        |  nodes,     |
                        |  namespaces,|
                        |  artifacts, |
                        |  networks,  |
                        |  instances, |
                        |  dnsmasq,   |
                        |  references,|
                        |  metadata,  |
                        |  cluster_   |
                        |  operations,|
                        |  work_queue,|
                        |  locks,     |
                        |  cluster    |
                        |  config,    |
                        |  events,    |
                        |  event_     |
                        |  objects,   |
                        |  node_      |
                        |  daemon_    |
                        |  states)    |
                        +-------------+

The database microservice (sf-database) centralizes all database access: - Only the database daemon has direct access to MariaDB - All other daemons use the gRPC interface - Provides Prometheus metrics for database operations

The sf-database box in the diagram represents a tier of N >= 1 replicas. All replicas connect to the same MariaDB; none is elected. Every other SF daemon reaches the tier through a client-side load-balanced gRPC channel constructed over the MARIADB_GATEWAY_HOSTS list of endpoints. Dead endpoints are skipped via subchannel connectivity state and client keepalives (10 s ping / 5 s timeout). sf-database also publishes the grpc.health.v1.Health protocol for external monitoring via unary Check calls; Watch-based client-side health checking is deliberately not enabled because the synchronous health servicer can deadlock the gRPC server's event thread (see shakenfist/util/grpc_channel.py). The overall ('') service status is dependency-aware: while sf-database is running it reports SERVING only while it can reach MariaDB, and flips to NOT_SERVING (on the ~10 s background loop) when MariaDB becomes unreachable. Schema currency is a refuse-to-start precondition enforced at startup, not a runtime health signal. See database.md — "MARIADB_HOST vs MARIADB_GATEWAY_HOSTS" — for the operator-facing detail.

Caller identity decides direct access

Going direct to MariaDB needs both MARIADB_HOST and a caller identity in mariadb.DIRECT_MARIADB_CALLERS (database and ctl). MARIADB_HOST alone is not enough, because it is rendered into /etc/sf/config, the systemd EnvironmentFile every daemon on a database-tier node shares. Only sf-database (which would otherwise call itself) and sf-ctl (which runs ensure-mariadb-schema and initialise-node before sf-database starts) act on it. Every other daemon uses the tier even when co-located with MariaDB: going direct would hide its load from the tier's metrics and connection accounting.

The identity is process-global, so an entry point calls set_caller_identity() (shakenfist/util/caller_identity.py) before anything that might dispatch. An unset identity reads as unknown, which routes an ordinary daemon to the tier correctly but would make sf-database call itself; sf-database therefore claims its identity as the first statement of main(), before write_pid_file() starts the eventlog drainer. The one exempt path is config.load_cluster_config(), which runs at import time before any identity exists -- see its docstring.

Schema management (ensure_schema() in mariadb.py, run via sf-ctl ensure-mariadb-schema) is version-gated per table, plus one un-gated pass: native MariaDB ENUM columns are reconciled against their Python enums on every run, because a MariaDB ENUM freezes its value list at CREATE TABLE time and a new Python enum member changes no table version. Without this pass, an upgrade adding an enum member (e.g. ObjectType.NAMESPACE_KEY) works on fresh installs but breaks existing databases with "Data truncated for column" errors. Enum columns are discovered from the SQLAlchemy metadata, so new sa.Enum(...) columns are covered automatically; the live upgrade path is exercised by the "Schema ENUM widening" CI job (tools/ci-enum-widening-test.sh).

Static object value cache

The database client in mariadb.py carries a small read-through cache for immutable static object values, restoring the etcd-era principle "objects are cacheable, attributes are not". Each public get_<type>() returns a frozen Pydantic model of static columns only (mutable fields — states, metadata, attributes, IPAM, daemon states — come from separate get_<type>_attributes() readers and are never cached). The cache sits above the direct/gRPC branch, so it serves both compute nodes (avoiding a gRPC round trip) and the sf-database daemon's own worker threads (avoiding a SQL query); it is a single process-global dict keyed (object_type, uuid) under a lock.

Correctness rests on four rules: only present rows are cached (never a miss, so a create-after-lookup or delete-then-lookup is never masked); every public update_<type>/delete_<type> evicts, and because the lazy online-upgrade persist routes through the public update_<type>, the cache self-heals after an upgrade; every entry is TTL-bounded, which is the only bound on staleness from a write made by another process; and a _db_get() copies any list or dict field it hands out, because frozen=True on the cached model stops attribute assignment rather than mutation of a container field's contents, so one reader mutating an uncopied list would be mutating every reader's — see "A frozen model is not a deep frozen model" in coding_rules.md. Two tiers set the TTL — OBJECT_CACHE_TTL_IMMUTABLE (default 300 s) for types whose static row changes only on create, delete, or a version-upgrade persist (instance, network, networkinterface, agentoperation, ipam) and OBJECT_CACHE_TTL_MUTABLE (default 30 s) for the upgradeable types (node, blob, artifact, upload, dnsmasq, namespace). Setting either to 0 disables that tier. Two of the immutable tier's members do have an update_<type> (update_ipam, update_network_interface); both evict, which is what keeps the tier correct, so an updater added to a type in this tier must evict too.

TTL bounds staleness, not residency. Expiry is lazy -- an entry is reclaimed only when its own key is read again -- so OBJECT_CACHE_MAX_ENTRIES (default 20000) bounds how much a process retains. Without it the cache holds every object a process has ever read rather than its working set, because an object read once and never read again is never revisited, and a delete in another process evicts nothing here. On overflow expired entries go first, then the entries closest to expiry.

Effectiveness is visible in the database_object_cache_{hits,misses, evictions}_total counters and in reduced database_get_<type>_total rates; database_object_cache_entries reports occupancy and database_object_cache_capacity_evictions_total reports pressure against the bound. Note that only sf-cluster, sf-resources and sf-database serve a metrics endpoint, so the client-side caches in sf-api, sf-net, sf-queues, sf-cleaner, sf-transfers and sf-sidechannel -- where most of the hit rate and most of the memory lives -- are not visible in Prometheus today.

SQL Filter-Pushdown Discipline

Object iteration uses one indexed SQL query per call rather than the older pattern of materialising all rows and filtering them in Python. Every find_artifacts, find_instances, find_networks, and find_network_interfaces call in shakenfist/mariadb.py JOINs the per-type static-values table to object_states and applies the caller's state, namespace, name and FK predicates directly in the WHERE clause. The two FK fields (network_uuid, instance_uuid) on ObjectFilterCriteria are honoured only by find_network_interfaces, which is what makes Network.networkinterfaces and Instance.interfaces query-backed properties returning hydrated NetworkInterface objects rather than the cached UUID lists they used to be.

The composite index idx_object_states_type_state on (object_type, state_value) covers the JOIN condition that is present in every query. Per-type name and namespace single-column indexes on the artifact, instance, and network tables cover the optional equality predicates. This keeps the common REST-layer calls — list-by-namespace, list-active, lookup-by-name — to an index scan with no full-table read.

Filter criteria are expressed as ObjectFilterCriteria in shakenfist/schema/object_filter.py. The iterator base class (DatabaseBackedObjectIterator in baseobject.py) builds criteria from its constructor arguments and delegates to the appropriate find_* primitive, so callers such as Artifacts(namespace=ns, prefilter='active') get SQL pushdown without any extra work at the call site. Filters that have no SQL equivalent (e.g. predicates over lazily-loaded attribute columns) remain as Python callables passed through the filters= argument and execute after the indexed scan.

See database.md — "SQL Filter Pushdown" — for per-API guidance and a code example.

gRPC Reliability

All gRPC calls use timeout=30 seconds. The _grpc_call() helper in mariadb.py enforces this for all database service calls and retries up to 3 times on UNAVAILABLE/DEADLINE_EXCEEDED errors with channel reset between attempts (wait_for_ready is deliberately left at the default of False so a wedged subchannel fails fast into this retry path instead of parking the caller). Once those retries are exhausted, _grpc_call() raises shakenfist.exceptions.DatabaseUnavailable rather than the underlying RpcError. The client wrappers in mariadb.py translate non-retryable RpcErrors into "object not found" return values (None/False/[]), but DatabaseUnavailable is deliberately not an RpcError subclass and propagates through them: an unreachable database must not be indistinguishable from a missing object. The few hot paths that intentionally tolerate an unreachable database catch it explicitly -- Daemon.check_daemon_state() skips the check, ClusterLock.__enter__ keeps retrying inside the caller's timeout, the queues daemon's health loop treats it as unhealthy and waits, and (since #3638) the cleaner's _maintain_blobs() and the cluster daemon's _cluster_wide_cleanup() and sweep helpers skip the affected work for the pass.

The servicer has the matching obligation. A reply whose only payload is a repeated field has nowhere to say "the read failed", so GetObjectsByState, GetStatelessObjectUuids and GetReferencesFrom must not answer a failed read with an empty list: when the direct accessor returns None (an OperationalError — MariaDB down, connection dropped, lock wait timeout, deadlock) they set UNAVAILABLE on the status, which _grpc_call retries and then surfaces as DatabaseUnavailable. An unexpected exception in the handler sets INTERNAL, which is non-retryable and so becomes a None return client-side. This matters because it is the failure mode where sf-database itself is healthy and answering, so nothing else in the stack notices.

get_active_blob_uuids() and get_node_blob_uuids() are the exceptions to the "wrappers translate to not found" rule: they raise DatabaseUnavailable rather than returning [], because the cleaner uses both lists as complement sets and unlinks every blob file named in neither. They are two accessors over reads that also have tolerant forms — get_objects_by_state() and get_references_from() — which keep collapsing a failed read for their iterate-only callers. Which form a call site needs is a property of the call site, so the pair exists rather than a flag on one accessor. See the "or [] is a decision" rule for how to decide which shape a new accessor should have, and PLAN-grpc-bounded-replies.md for the reply-size work this came out of.

The object iterators are the widest consumer of all this, and they propagate. DatabaseBackedObjectIterator._find() catches nothing: a DatabaseUnavailable raised part way through Blobs(), IPAMs() or AgentOperations() unwinds into the caller mid-iteration rather than ending the loop quietly, because a read that did not happen must not present as an iteration that found nothing. Daemon loops which tolerate that catch it at the top of the pass, as the cluster daemon's cleanup does; a new caller which cannot tolerate a partially consumed iterator has to say so. The None failure shape does still truncate there, which is safe only while every iterator caller iterates the result rather than complementing it — that is recorded, with the rest of the audit, in the plan.

The database gRPC channel uses HTTP/2 keepalive (ping every 10s, 5s timeout) to detect stale connections before they cause failures, and a 32MiB client receive cap (raised from grpcio's 4MiB default in #3638, where GetObjectEvents and GetObjectsByState replies outgrew it and failed as RESOURCE_EXHAUSTED). The cap is client-side only; the server sets no send cap, so an oversized reply is still serialised before it fails, which is why the durable fix is bounding replies rather than raising the cap again -- see PLAN-grpc-bounded-replies.md. The database gRPC server uses a 64-thread pool to handle concurrent requests from all daemons. The database client in mariadb.py (_grpc_call) retries UNAVAILABLE and DEADLINE_EXCEEDED failures, rebuilding the channel on a wedged subchannel but keeping it on a refused connection so round_robin can serve the retry from a surviving gateway. All gRPC failures are logged at ERROR level.

get_objects_by_state() returns None on non-retryable errors (distinct from [] for no matches). All object iterators handle this by falling back to unfiltered scans, ensuring that such failures do not silently drop objects from iteration results (e.g. interfaces during instance deletion). A database outage instead raises DatabaseUnavailable out of the iterator.

Object iterators read from MariaDB via get_all_*() functions. All object static values now live in MariaDB.

Cluster Operation Tracking

Every operation schema's model class declares its target objects via a target_fields: ClassVar[dict[str, ObjectType]] class variable. When enqueue_cluster_operation (in schema/operations/util.py) writes the cluster_operations row, it reads that declaration and writes one cluster_operation_targets row per non-None target field. Callers have no per-target bookkeeping obligation — targets are recorded automatically.

Each target row also carries an observational flag (schema v3), derived at enqueue time by is_observational() in schema/operations/util.py. A model may declare observational_tasks: ClassVar[frozenset[...]] beside target_fields; an operation is observational only if it has tasks and every one of them is in that set, so a single mutating task makes the whole operation mutating. Three tasks on two models are declared today: collect_billing_statistics and health_check_kvm_process on node_inst_op, and verify_size_and_checksum on node_blob_op. The instance sweep enqueues the first two together, and every caller enqueues the third on its own. Every other task mutates. The rationale is in the next section.

When is a task observational?

A task is observational if its purpose is to measure or verify its target, and its only writes are of one of two kinds:

  • Recording what it observed. Events, usage records and computed hashes are the examples. No other operation is sequenced behind these writes. A user can wait for one: GET /blobs/<uuid>/checksum/<algorithm> enqueues verify_size_and_checksum when the hash is not yet known, and returns it once recorded. That caller polls for the value, though, not for the operation, so it does not need the operation to count as pending.
  • Correcting a cached observation that reality has already falsified. health_check_kvm_process clears an instance's kvm_pid only when that process no longer exists, and verify_size_and_checksum drops a replica location only when the file is proven missing or corrupt. In both cases the task reveals a change, it does not make one.

Anything that changes desired state, guest-visible state, or something a user action or another operation depends on is mutating. That includes creating, starting, deleting and snapshotting, routing, DHCP and NAT changes, and replicating a blob onto a node.

When in doubt, a task is mutating. The two mistakes are not equally costly. Marking a mutating task observational hides it from last_cluster_operation and has_pending_cluster_operation(), which are the gates that exist to wait for it, and that is the very bug the flag was introduced to fix. Marking an observational task mutating costs only a stale pointer, because a sweep briefly becomes the object's latest operation.

Two practical traps:

  • The declaration must be annotated ClassVar. Without it pydantic turns the attribute into a model field, and is_observational() silently sees an empty set, so every operation of the model is stored as mutating with no error anywhere.
  • Marking a task observational also changes what a failing task does. When a node_inst_op task raises, NodeInstOp.dispatch_task() marks the operation errored, and for a mutating operation it moves the instance to an error state too. For an observational operation it leaves the instance alone, because a failure to measure a guest says nothing about the guest. The question is asked of the whole operation, with is_observational(), not of the task which failed: an operation which runs a probe ahead of a mutating task is mutating, and a failed probe ends it before the mutating task runs, so the instance is errored. So a mutating task marked observational would also stop erroring its instance when it fails.

The classification of every task is pinned by shakenfist/tests/schema/operations/test_observational_tasks.py. It holds a table of every task of every operation model, so adding a task or a model fails the test until someone classifies it. Update the table in the same change that adds the task, and add the task to observational_tasks only if it is observational. The criteria above say why each existing task is classified as it is.

DatabaseBackedObjectWithOperations exposes three read shapes over the history:

  • last_cluster_operation (property): returns the most recent non-observational target row regardless of its state. Consumed by external_view() projections and runs_after=[...] chains, which want the latest pointer independent of whether it has reached a terminal state.
  • has_pending_cluster_operation() (method): returns True if any non-observational target row's operation is in {queued, preflight, executing}, the ACTIVE_OPERATION_STATES set in schema/operations/baseclusteroperation.py which every reader of the in-flight and terminal sets shares. Consumed by Network.is_okay() and any future history-aware gate. The query joins cluster_operation_targets against object_states, so a later terminal operation cannot mask an earlier in-flight one. external_view() on instances, networks and artifacts renders this as has_pending_operations. It is false for an in-memory object and true when the database cannot answer.
  • get_cluster_operations() (method): returns the target rows for the object, newest first by sequence number, each hydrated into its operation and paired with the observational flag stored on its row. By default the rows are filtered in SQL to operations which are pending, using the same join as has_pending_cluster_operation(). The two do not always agree: the listing is not filtered on the observational flag, so it includes pending observational operations, each reported with observational: true, while has_pending_cluster_operation() ignores them. It does not follow depends_on or runs_after, so operations on other objects are not included. The per-object /clusteroperations endpoints render it, and Instance.enqueue_delete() walks it to decide what to abort.

As a backstop for rows that outlive their operation, the table is bounded by a periodic prune in the cluster daemon, alongside the existing delete_stale_transfers cleanup. The prune removes rows older than CLUSTER_OPERATION_TARGET_RETENTION seconds whose operation has already reached a terminal state. In-flight operations (queued/preflight/executing) are never pruned regardless of age. Because the cluster daemon already runs cluster-wide cleanup under ClusterLock election, no additional locking or master-node gating is required.

Only these two readers ignore observational rows. The other by-target readers deliberately see everything:

  • get_cluster_operation_targets_for_object(), which both the per-blob sweep (to avoid queueing a duplicate checksum op) and get_cluster_operations() (the per-object listings) use;
  • the by-target listing;
  • get_recent_terminal_op_states_for_target(), a circuit breaker which wants every failure of a network operation, none of which is observational.

Rows normally disappear when their operation is hard-deleted (eligible for removal thirty seconds after it goes terminal and removed by the deleted-object sweep), so the prune is a backstop.

Cluster Operation Storage and Work Queues

Cluster operation headers and the per-node work queues both live in MariaDB. The cluster_operations table stores the full operation metadata as JSON in metadata_json, with node_uuid, instance_uuid, network_uuid and priority extracted into indexed columns for dispatch-time filtering. The work_queue table stores one row per queued job with claim fields (claimed_at, claimed_by, attempts) on the same row -- MariaDB row locking replaces the old etcd two-prefix (/sf/queue/, /sf/processing/) design. Dequeue uses SELECT ... FOR UPDATE SKIP LOCKED so concurrent workers either claim distinct rows or one gets nothing.

Creating a cluster operation is atomic: the CreateAndEnqueueCluster gRPC RPC writes the cluster_operations row, the object_states row, and the work_queue row in a single MariaDB transaction. Audit events are written directly into MariaDB via the local spool drainer's mariadb.record_event_batch call.

The cluster daemon runs reap_stuck_cluster_operation_jobs() from daemons/cluster/scheduled_tasks.py on a one-minute schedule. For every row whose claimed_at is older than CLUSTER_OP_STUCK_THRESHOLD seconds, the reaper either clears the claim so a fresh worker picks the job up or -- if attempts has reached CLUSTER_OP_MAX_ATTEMPTS -- deletes the row and transitions the underlying cluster operation to STATE_ERROR. A second pass in the same tick sweeps the inverse shape: cluster operations sitting in queued or executing for longer than the same threshold with no work_queue row referencing them at all (failure debris -- the enqueue writes both in one transaction, so the shape cannot occur in normal operation). Nothing can ever dequeue such an operation, so the pass transitions it to STATE_ERROR with an audit event; the enqueue-side coalescing dedup independently refuses to return a rowless op, so a caller's "ensure" enqueues fresh work rather than adopting a dead op (issue 4303). Reaper activity is exported on cluster_op_reaper_requeued_total, cluster_op_reaper_rejected_total and cluster_op_reaper_orphaned_total, scraped from CLUSTER_METRICS_PORT on the cluster daemon.

The reaper is the slow path. A worker which is still running when its sf-queues stops -- SIGKILLed at the end of TimeoutStopSec, abandoned when the worker pool's 20 second stop grace (WORKER_STOP_GRACE in daemons/daemon.py) runs out, or lost in a crash -- leaves its operation executing and its row claimed. Clearing the claim alone does not recover it: the next worker to dequeue the row finds the operation executing, skips it and resolves the row, which produces exactly the rowless shape the orphan pass sweeps, 30 minutes later (issue 4543). So the restarted sf-queues, in startup_tasks, first looks at the claimed rows on its own node's queues (named by node uuid): nothing else consumes those queues and it has started no workers yet, so an operation still executing behind one was being run by the process it replaces. It moves the operation back to queued -- the same transition a task's defer_with_backoff() makes -- so the row re-runs it from the first task once the claims are cleared, or, if the row has been claimed CLUSTER_OP_MAX_ATTEMPTS times, deletes the row and errors the operation as the reaper would. Rows on the shared any-* queues are left to the reaper, because nothing records which node claimed them: over gRPC, claimed_by is the database daemon's node name, not the caller's. For the same reason startup clears the claims on its own queues only. A claim on an any-* queue may be another node's worker running that item right now, and clearing it would hand the item to a second worker. Recovery is best effort, one item at a time. An item it cannot recover, for example because its operation type no longer exists, is logged and left for the reaper. If the claims cannot be listed at all, startup clears none of them, because clearing a claim nobody has checked strands its operation just as before.

The cluster daemon also runs two other reaping sweeps from daemons/cluster/scheduled_tasks.py. per_deleted_object_checks() (every minute, within DELETED_OBJECT_SWEEP_SECONDS) hard deletes objects that have been in a final state (deleted, complete, abort) for longer than their grace period; it holds a list of uuids per object type, filled a page at a time with the age filter pushed down to SQL, and takes one object from each type in turn. Objects are hydrated one at a time at processing time inside a per-item exception guard. reconcile_orphaned_objects() (hourly) removes "phantom" object_states rows whose static-values row is gone (with an age guard so mid-creation objects are never raced) and repairs "zombie" static rows that have no state row by writing a state row once the zombie has been seen on two consecutive sweeps; node and namespace objects are excluded from zombie repair. The state written comes from zombie_repair_state(), which defaults to deleted; NamespaceKey, TrustedIssuer and MappingRule override it to return created when their attributes row exists, since their authentication read path honours them without a state row. A failed liveness read (DatabaseUnavailable) writes nothing and ends the pass. Both kinds of orphan are otherwise invisible to every state-driven iterator.

The elected cluster node also runs reconcile_scheduler_capacity() every five minutes. That cadence is anchored to a cluster-wide last-run stamp (the SCHEDULED_TASK_LAST_RUN_RECONCILE_SCHEDULER_CAPACITY key in cluster_config) rather than to process start, and the elected maintainer's own maintenance pass forces a reconcile as soon as it sees an active hypervisor with fresh metrics and no scheduler_node_capacity row, instead of waiting out the cadence. Both exist because the reconciler is the only thing which creates capacity rows, and a node without a row is admitted against nothing at all: a process-local five minute timer left every placement in a new cluster's first minutes unguarded, and the first pass then recorded the resulting over-limit usage on the row it created (issue 4087). The check used to be a one-shot on the election path, which is why it missed a cold cluster entirely -- election happens seconds after start-up, before any hypervisor has published, so the pass it forced had nothing to size. See the subsystem internals for the predicate it applies now. One pass is a single ReconcileSchedulerCapacity RPC which expires stale namespace claims, re-derives per-hypervisor limits from the typed node_metrics columns, recomputes usage counters from placed instances and the decaying expected-demand signal, and rebuilds the cluster_capacity singleton. The reconciler recomputes the three capacity tables (scheduler_node_capacity, namespace_claims, cluster_capacity) wholesale; the atomic admission and release RPCs also write them incrementally, and are the sole drawdown path against them, so a divergence between what the reconciler computes and what the counters hold is drift, healed on the next pass rather than expected steady state. The claim CRUD RPCs are a third writer: they move capacity between namespace_claims and cluster_capacity (a claim's limits into claimed_*, its namespace's existing drawdown out of unclaimed_used_* and onto the claim, and the reverse on deletion) but consume none, so the same drift-healing property holds. See subsystem internals. Observability is the scheduler_capacity_* gauges and counters exported on CLUSTER_METRICS_PORT, plus one structured log line per pass.

Batched, Priority-Aware Dequeue

sf-net and the sf-queues worker pool both call a single mariadb.dequeue_work_items(queue_names, limit) primitive. The caller passes the queue names in priority order (index 0 = top priority); MariaDB honours that order via ORDER BY FIELD(queue_name, ...), scheduled_at so one SELECT returns the most important eligible work first across an arbitrary number of queues. The previous one-RPC-per-queue polling loop is replaced by one RPC per iteration regardless of how many priority lanes the worker drains.

The lanes, from PRIORITY in shakenfist/schema/operations/baseclusteroperation.py, highest first: user_waiting (a user is blocked on the result, so latency matters most), user_facing (ordinary user-initiated work), background (maintenance nobody is waiting for), and user_facing_high_io / background_high_io, which carry the same meaning for disk-heavy work. A worker skips the high_io lanes while its node's disk is more than about 80% busy (see shakenfist/daemons/daemon.py), so that work waits rather than piling onto a saturated disk. Each lane exists both per node and globally.

Lower-priority rows only spill in when the higher-priority queues yield fewer rows than limit, so sustained heavy load on user_facing can still crowd background out -- explicit fairness (bounded staleness or reserved-slot) is intentionally deferred to a follow-up step. Worker crash recovery is unchanged: any claimed- but-not-yet-executed rows that the worker doesn't run are picked up by the stuck-row reaper described above.

To see what that costs an operation in practice, tools/queue-wait-report.py reports the queue-wait distribution by queue class, priority lane and operation type, and tools/operation-timeline.py splits one operation's wait into the initial queue sit, the deferral the dispatcher asked for and the sit after each redelivery -- which is what distinguishes a lane that is genuinely backed up from one whose operations are waiting on a dependency. Both are documented, with their caveats, under networking observability, where they sit next to the other queue tooling. Their parsers are checked against the emitter by shakenfist/tests/test_queue_tooling_round_trip.py, which has a real NetOp emit each event through the real event echo and the real JSON formatter, so renaming a field in execution_duration_extra() or defer() fails the suite rather than quietly emptying a report.

Coalescible Operations

Some operation tasks are idempotent reconciliation work whose effect depends only on current DB state, not on the count of pending ops asking for it. The canonical example is network_apply_update_dnsmasq: six instance starts on the same network each enqueue one, but the resulting dnsmasq config covers every lease no matter whether the worker ran it once or six times.

Op classes that have such tasks declare them on the class:

class NetOp(BaseClusterOperation):
    coalescible_tasks = schema.COALESCIBLE_TASKS
    coalescible_key_columns = ('network_uuid', 'node_uuid')

coalescible_key_columns names a tuple of indexed columns on cluster_operations, not a single one: a sibling has to match every column before it counts as the same work, so the key can be narrower than "the same network" without any query rewriting. A column with no value on the operation binds IS NULL rather than being dropped from the comparison -- that distinction matters here. NetOp's two cluster-wide tasks (network_apply_update_dnsmasq, network_apply_create_network_node) carry node_uuid = None, so they fold only each other; network_ensure_mesh, enqueued per participating hypervisor, carries its own node's uuid and folds only that node's own siblings. Both are strictly narrower than the network-alone key this replaced, which is the property the whole widening rests on. None is bound as IS NULL rather than refused: refusing a None key value would have silently switched off the only coalescing the cluster does the moment the key grew a second column.

The fold runs at two layers, both controlled by this metadata:

  • Enqueue-side dedup (mariadb.find_existing_coalescible_op): create_and_enqueue in the schema module checks for an existing pending single-task coalescible op matching every key column before inserting a new row. If found, the new caller's op_uuid is the existing op's op_uuid. All raise_for_error waiters then block on the same op and the worker runs it once.

  • Worker-side fold (mariadb.fold_coalescible_siblings): inside BaseClusterOperation.execute, the survivor atomically moves every other pending coalescible op matching the same key from queued to executing in one SQL statement, with the state message coalesced into <survivor uuid>. Only once every one of the survivor's own tasks has succeeded does it move those siblings on to complete. A 'coalesced sibling ops' event on the survivor records the folded uuids.

The order matters. A sibling is often an operation something else waits on: an instance create names the per-node network_ensure_mesh in depends_on, and an API caller blocks in raise_for_error(). Until #4638 the fold set siblings complete before the survivor had run anything, so those waiters were released as soon as the survivor started, and went ahead even if it then failed or deferred. Now a sibling reads as in progress until the work is actually done.

On every other survivor outcome -- an error, a defer back to queued, an exception escaping execute(), an abort, or a crash -- the siblings are simply left executing. Nothing is propagated and nothing has to be undone. The fold never touches work_queue, so each sibling still has its own row, and when that row is dequeued the sf-net dispatcher runs the sibling itself: it skips only terminal states, and a transition to executing from executing is a no-op. A sibling's final state is therefore always its own real outcome, and a dependent aborts on its error exactly as it would on any other dependency's. The same holds if completing a sibling fails (the database is unavailable, say): the failure is logged rather than raised, so the survivor's work item still resolves, and the sibling runs itself.

Two things have to stay true for that to work, and the PARTITIONED-WORKER SAFETY INVARIANT comment in shakenfist/daemons/network/workitem.py states both. sf-net must keep running an operation it dequeues in executing, and every foldable operation must be drained by sf-net and never by sf-queues, which drops anything it dequeues that is not queued -- a folded sibling which reached it would be stranded until the orphan reaper errored it. The two guards described below are what keep foldable work off sf-queues.

The enqueue-side dedup is the cheaper of the two -- the row never gets inserted -- but the worker-side fold is the safety net for the race where two concurrent callers both lose the lookup.

The enqueue-side dedup will not adopt a slower lane

Reuse is one-sided. create_and_enqueue passes find_existing_coalescible_op the list of priorities it is willing to adopt, which is every PRIORITY at least as urgent as its own. Adopting a more urgent pending op costs the caller nothing: the work runs sooner than it asked for. Adopting a less urgent one means adopting its queue -- the name is {target}-{family}-{priority} -- so the caller sitting in raise_for_error(), and any runs_after dependency hanging off the returned op, wait out the slower lane's queue-sit tail instead of their own.

That overlap is not hypothetical for either coalescible task with more than one enqueue site. Network.ensure_mesh enqueues network_ensure_mesh at user_facing while the network maintainer enqueues the same task, for the same network and node, at background -- and it does so precisely when is_mesh_okay() reports drift, which is the state a network is in immediately after an instance starts elsewhere on it. create_on_network_node and the maintainer straddle user_facing/background for network_apply_create_network_node in the same way, and have done since before the mesh task became coalescible.

The worker-side fold needs no equivalent rule. It runs at dispatch, on an operation which has already been dequeued, so a survivor at any priority is about to do the work immediately; the queue a folded sibling was sitting on no longer matters.

The enqueue-side dedup will not adopt an op which is backing off

Nor will it adopt a queued op whose work_queue row is scheduled more than COALESCIBLE_ADOPT_SCHEDULE_MARGIN_SECONDS into the future. A queued op with a row due later than that is backing off after a defer: it is still live, so the orphan guard above does not exclude it, but adopting it hands the caller the 15, 30 or 60 second delay the defer asked for, and the spent defer_count with it, when a fresh op would have run at once (#4643). The margin is the dispatcher's idle poll, IDLE_POLL_MAX_SECONDS, because a row due within it runs no later than a fresh row would. Nothing is lost either way: if the lookup declines, a new op is inserted, and the worker-side fold catches the duplicate once both are dequeued. The orphan reaper passes no bound, because a row scheduled for later is still a row.

Both guards are key-aware and family-aware

cluster_operations has no queue column, so the key is the only thing either primitive's SQL can use to confine a fold to the operations one dispatcher process actually drains. But naming node_uuid in the key is necessary, not sufficient: which dispatcher drains a per-node operation also depends on the queue family the enqueue chose, because enqueue_cluster_operation builds the queue name as {target}-{family}-{priority}. A node target only reaches sf-net's per-node {node}-network-* queue when the caller also passes family='network'; the default clusteroperation family routes the same target to sf-queues instead, which fills spare worker slots from a single unpartitioned dequeue call and has no per-target routing at all. sf-net, by contrast, hashes every operation for a given target to one worker thread (_routing_key in shakenfist/daemons/network/workitem.py), so a fold there never claims an operation another thread is running, and a survivor never completes one. That guarantee is the load-bearing part of the safety argument for a per-node fold, and it does not extend to sf-queues -- see the PARTITIONED-WORKER SAFETY INVARIANT comment at the top of Job.__init__ in that file for the argument in full, including why NodeNetOp.network_apply_create_hypervisor (whose model already carries both columns) stays out of COALESCIBLE_TASKS.

Both guards therefore test the family as well as the key:

  • The enqueue-time guard (InvalidCoalescibleEnqueue in shakenfist/schema/operations/net_op.py) refuses a coalescible task enqueued to a non-networknode target unless node_uuid is in coalescible_key_columns, the operation's derived node_uuid is set, and the enqueue is on the network family.
  • The fold-time guard (BaseClusterOperation.execute) admits a per-node queue on the same condition, read off the queue name's {target}-{family}- prefix, and records the outcome key_cannot_distinguish_queue when it refuses. That name replaces not_cluster_wide; tools/queue-wait-report.py still maps the old name out of retained log history rather than dropping it, since it was the recorded outcome for every build before the key became multi-column.

Two further outcomes exist so that a fold which did not happen is never reported as one which happened and matched nothing -- the reading that let #3878 sit undetected for three months:

  • key_column_missing: a key column which is not an attribute of the operation class at all. That is a mis-declaration, and it is a different thing from a column which is present and None. None binds IS NULL, which is how a cluster-wide NetOp folds only the other cluster-wide operations on its network. A column the class does not have is NULL on every row too, so binding IS NULL for it would quietly degenerate the key to whichever columns did resolve -- for a NetOp, back to the network alone, which is the cross-node fold this design exists to prevent. The fold therefore tests hasattr and skips, rather than reading the value with a default.
  • coalescing_unavailable: the database service answered UNIMPLEMENTED, meaning a rolling upgrade against an sf-database predating the RPC the client called -- FoldCoalescibleSiblings for the fold, FindExistingCoalescibleOpV2 for the dedup. The wrappers raise exceptions.CoalescingUnavailable for this rather than returning the same "nothing matched" they return on every other failure, so an operator can watch the upgrade window open and close in the same table as everything else instead of grepping daemon logs.

The V2 gRPC methods

ClaimCoalescibleSiblingsV2 and FindExistingCoalescibleOpV2 take a repeated CoalescibleKeyPair keys. They carry the suffix because they replaced a single-column pair (ClaimCoalescibleSiblings, FindExistingCoalescibleOp, with a scalar target_column and target_uuid) as new RPC names rather than a new field on the existing messages. A repeated field would be wire-compatible but unsafe across a rolling upgrade: an old sf-database would silently ignore the extra columns and fold on the network alone, which is exactly the cross-node corruption the wider key exists to prevent. An old server answers a V2 call with UNIMPLEMENTED, and the client treats that as "coalescing unavailable" and skips the fold -- a loud, temporary loss of an optimisation instead of a quiet correctness failure. "Loud" means the coalescing_unavailable outcome above, not merely a log line. The single-column pair was removed in #4639: no release ever carried it (every tagged release predates NetOp), so no client can still send it, and its handlers accepted any task name with a single key column, which is the cross-node fold the V2 names exist to refuse. There is nothing for a client to fall back to, and shakenfist/tests/test_mariadb_coalescing.py checks that stays so.

FindExistingCoalescibleOpV2 additionally carries repeated string priorities, the priority filter described above. Unlike a key column, an old V2 server ignoring that field is only a lost optimisation -- it deduplicates the way the code did before the field existed -- so it travels as a field rather than needing a V3.

FoldCoalescibleSiblings takes the same request and returns the same reply as ClaimCoalescibleSiblingsV2, and runs the same statement, but moves the siblings to executing rather than complete. It is a new RPC rather than a flag on the request for the same reason the V2 pair is: an old server would ignore the flag and complete the siblings up front. A client which gets UNIMPLEMENTED from it skips the fold and never falls back to ClaimCoalescibleSiblingsV2, which would bring back exactly the premature completion it replaces. ClaimCoalescibleSiblingsV2 keeps its old behaviour for clients which predate the fold and has no caller in this tree. No release carried it either, so it can be retired the way the single-column pair was.

Batched hard deletes

The deleted-object sweep hard deletes most object types through the HardDeleteObjects RPC, which deletes up to 1000 objects of one type in a single transaction. A call is either guarded, naming the states and the age an object must still have, with the check made in the same transaction as the delete, or unconditional, for a caller which has already decided (this is what DatabaseBackedObject.hard_delete() sends for one object). The rows a type loses are declared in one place, HARD_DELETE_BATCH_FUNCTIONS in mariadb.py, which lists the public delete functions run for the type before delete_state, delete_object_metadata and delete_object_events; HARD_DELETE_STATEMENTS maps each name to its many-object statement and to a call of the public function. Instances, namespaces, namespace claims and nodes are absent because their hard delete does more than delete rows. An sf-database which predates the RPC makes hard_delete_objects() raise HardDeleteBatchUnavailable, and callers fall back to hard_delete_objects_per_call(), the same deletes as separate commits. The client remembers that answer for HARD_DELETE_BATCH_RECHECK_SECONDS (five minutes) and raises without asking again until then, so a rolling upgrade does not cost every hard delete a refused round trip and a warning. shakenfist/tests/test_hard_delete_batch_parity.py checks the mapping against each class's own deletes, so its EXPECTED_DELETES literal must change whenever the mapping does.

📝 Report an issue with this page