Skip to content

Database internals

How Shaken Fist uses MariaDB beyond the deployment and schema concerns covered in the operator guide: the static object value cache, filter-pushdown discipline, gRPC reliability, and the cluster operation tracking and work-queue machinery.

+----------------+     +----------------+     +----------------+
|   sf-api       |     |   sf-cleaner   |     |   sf-queues    |
|   sf-net       |     |   sf-cluster   |     |   sf-resources |
+-------+--------+     +-------+--------+     +-------+--------+
        |                      |                      |
        +----------------------+----------------------+
                               |
                         gRPC (13005)
                               |
                    +----------+----------+
                    |    sf-database      |
                    +----------+----------+
                               |
                        +------+------+
                        |   MariaDB   |
                        | (state,     |
                        |  IPAM,      |
                        |  uploads,   |
                        |  blobs,     |
                        |  nodes,     |
                        |  namespaces,|
                        |  artifacts, |
                        |  networks,  |
                        |  instances, |
                        |  dnsmasq,   |
                        |  references,|
                        |  metadata,  |
                        |  cluster_   |
                        |  operations,|
                        |  work_queue,|
                        |  locks,     |
                        |  cluster    |
                        |  config,    |
                        |  events,    |
                        |  event_     |
                        |  objects,   |
                        |  node_      |
                        |  daemon_    |
                        |  states)    |
                        +-------------+

The database microservice (sf-database) centralizes all database access: - Only the database daemon has direct access to MariaDB - All other daemons use the gRPC interface - Provides Prometheus metrics for database operations

The sf-database box in the diagram represents a tier of N >= 1 replicas. All replicas connect to the same MariaDB; none is elected. Every other SF daemon reaches the tier through a client-side load-balanced gRPC channel constructed over the MARIADB_GATEWAY_HOSTS list of endpoints. Dead endpoints are skipped via subchannel connectivity state and client keepalives (10 s ping / 5 s timeout). sf-database also publishes the grpc.health.v1.Health protocol for external monitoring via unary Check calls; Watch-based client-side health checking is deliberately not enabled because the synchronous health servicer can deadlock the gRPC server's event thread (see shakenfist/util/grpc_channel.py). The overall ('') service status is dependency-aware: while sf-database is running it reports SERVING only while it can reach MariaDB, and flips to NOT_SERVING (on the ~10 s background loop) when MariaDB becomes unreachable. Schema currency is a refuse-to-start precondition enforced at startup, not a runtime health signal. See database.md — "MARIADB_HOST vs MARIADB_GATEWAY_HOSTS" — for the operator-facing detail.

Schema management (ensure_schema() in mariadb.py, run via sf-ctl ensure-mariadb-schema) is version-gated per table, plus one un-gated pass: native MariaDB ENUM columns are reconciled against their Python enums on every run, because a MariaDB ENUM freezes its value list at CREATE TABLE time and a new Python enum member changes no table version. Without this pass, an upgrade adding an enum member (e.g. ObjectType.NAMESPACE_KEY) works on fresh installs but breaks existing databases with "Data truncated for column" errors. Enum columns are discovered from the SQLAlchemy metadata, so new sa.Enum(...) columns are covered automatically; the live upgrade path is exercised by the "Schema ENUM widening" CI job (tools/ci-enum-widening-test.sh).

Static object value cache

The database client in mariadb.py carries a small read-through cache for immutable static object values, restoring the etcd-era principle "objects are cacheable, attributes are not". Each public get_<type>() returns a frozen Pydantic model of static columns only (mutable fields — states, metadata, attributes, IPAM, daemon states — come from separate get_<type>_attributes() readers and are never cached). The cache sits above the direct/gRPC branch, so it serves both compute nodes (avoiding a gRPC round trip) and the sf-database daemon's own worker threads (avoiding a SQL query); it is a single process-global dict keyed (object_type, uuid) under a lock.

Correctness rests on four rules: only present rows are cached (never a miss, so a create-after-lookup or delete-then-lookup is never masked); every public update_<type>/delete_<type> evicts, and because the lazy online-upgrade persist routes through the public update_<type>, the cache self-heals after an upgrade; every entry is TTL-bounded, which is the only bound on staleness from a write made by another process; and a _db_get() copies any list or dict field it hands out, because frozen=True on the cached model stops attribute assignment rather than mutation of a container field's contents, so one reader mutating an uncopied list would be mutating every reader's — see "A frozen model is not a deep frozen model" in coding_rules.md. Two tiers set the TTL — OBJECT_CACHE_TTL_IMMUTABLE (default 300 s) for types whose static row changes only on create, delete, or a version-upgrade persist (instance, network, networkinterface, agentoperation, ipam) and OBJECT_CACHE_TTL_MUTABLE (default 30 s) for the upgradeable types (node, blob, artifact, upload, dnsmasq, namespace). Setting either to 0 disables that tier. Two of the immutable tier's members do have an update_<type> (update_ipam, update_network_interface); both evict, which is what keeps the tier correct, so an updater added to a type in this tier must evict too.

TTL bounds staleness, not residency. Expiry is lazy -- an entry is reclaimed only when its own key is read again -- so OBJECT_CACHE_MAX_ENTRIES (default 20000) bounds how much a process retains. Without it the cache holds every object a process has ever read rather than its working set, because an object read once and never read again is never revisited, and a delete in another process evicts nothing here. On overflow expired entries go first, then the entries closest to expiry.

Effectiveness is visible in the database_object_cache_{hits,misses, evictions}_total counters and in reduced database_get_<type>_total rates; database_object_cache_entries reports occupancy and database_object_cache_capacity_evictions_total reports pressure against the bound. Note that only sf-cluster, sf-resources and sf-database serve a metrics endpoint, so the client-side caches in sf-api, sf-net, sf-queues, sf-cleaner, sf-transfers and sf-sidechannel -- where most of the hit rate and most of the memory lives -- are not visible in Prometheus today.

SQL Filter-Pushdown Discipline

Object iteration uses one indexed SQL query per call rather than the older pattern of materialising all rows and filtering them in Python. Every find_artifacts, find_instances, find_networks, and find_network_interfaces call in shakenfist/mariadb.py JOINs the per-type static-values table to object_states and applies the caller's state, namespace, name and FK predicates directly in the WHERE clause. The two FK fields (network_uuid, instance_uuid) on ObjectFilterCriteria are honoured only by find_network_interfaces, which is what makes Network.networkinterfaces and Instance.interfaces query-backed properties returning hydrated NetworkInterface objects rather than the cached UUID lists they used to be.

The composite index idx_object_states_type_state on (object_type, state_value) covers the JOIN condition that is present in every query. Per-type name and namespace single-column indexes on the artifact, instance, and network tables cover the optional equality predicates. This keeps the common REST-layer calls — list-by-namespace, list-active, lookup-by-name — to an index scan with no full-table read.

Filter criteria are expressed as ObjectFilterCriteria in shakenfist/schema/object_filter.py. The iterator base class (DatabaseBackedObjectIterator in baseobject.py) builds criteria from its constructor arguments and delegates to the appropriate find_* primitive, so callers such as Artifacts(namespace=ns, prefilter='active') get SQL pushdown without any extra work at the call site. Filters that have no SQL equivalent (e.g. predicates over lazily-loaded attribute columns) remain as Python callables passed through the filters= argument and execute after the indexed scan.

See database.md — "SQL Filter Pushdown" — for per-API guidance and a code example.

gRPC Reliability

All gRPC calls use timeout=30 seconds. The _grpc_call() helper in mariadb.py enforces this for all database service calls and retries up to 3 times on UNAVAILABLE/DEADLINE_EXCEEDED errors with channel reset between attempts (wait_for_ready is deliberately left at the default of False so a wedged subchannel fails fast into this retry path instead of parking the caller). Once those retries are exhausted, _grpc_call() raises shakenfist.exceptions.DatabaseUnavailable rather than the underlying RpcError. The client wrappers in mariadb.py translate non-retryable RpcErrors into "object not found" return values (None/False/[]), but DatabaseUnavailable is deliberately not an RpcError subclass and propagates through them: an unreachable database must not be indistinguishable from a missing object. The few hot paths that intentionally tolerate an unreachable database catch it explicitly -- Daemon.check_daemon_state() skips the check, ClusterLock.__enter__ keeps retrying inside the caller's timeout, the queues daemon's health loop treats it as unhealthy and waits, and (since #3638) the cleaner's _maintain_blobs() and the cluster daemon's _cluster_wide_cleanup() and sweep helpers skip the affected work for the pass.

The servicer has the matching obligation. A reply whose only payload is a repeated field has nowhere to say "the read failed", so GetObjectsByState, GetStatelessObjectUuids and GetReferencesFrom must not answer a failed read with an empty list: when the direct accessor returns None (an OperationalError — MariaDB down, connection dropped, lock wait timeout, deadlock) they set UNAVAILABLE on the status, which _grpc_call retries and then surfaces as DatabaseUnavailable. An unexpected exception in the handler sets INTERNAL, which is non-retryable and so becomes a None return client-side. This matters because it is the failure mode where sf-database itself is healthy and answering, so nothing else in the stack notices.

get_active_blob_uuids() and get_node_blob_uuids() are the exceptions to the "wrappers translate to not found" rule: they raise DatabaseUnavailable rather than returning [], because the cleaner uses both lists as complement sets and unlinks every blob file named in neither. They are two accessors over reads that also have tolerant forms — get_objects_by_state() and get_references_from() — which keep collapsing a failed read for their iterate-only callers. Which form a call site needs is a property of the call site, so the pair exists rather than a flag on one accessor. See the "or [] is a decision" rule for how to decide which shape a new accessor should have, and PLAN-grpc-bounded-replies.md for the reply-size work this came out of.

The object iterators are the widest consumer of all this, and they propagate. DatabaseBackedObjectIterator._find() catches nothing: a DatabaseUnavailable raised part way through Blobs(), IPAMs() or AgentOperations() unwinds into the caller mid-iteration rather than ending the loop quietly, because a read that did not happen must not present as an iteration that found nothing. Daemon loops which tolerate that catch it at the top of the pass, as the cluster daemon's cleanup does; a new caller which cannot tolerate a partially consumed iterator has to say so. The None failure shape does still truncate there, which is safe only while every iterator caller iterates the result rather than complementing it — that is recorded, with the rest of the audit, in the plan.

The database gRPC channel uses HTTP/2 keepalive (ping every 10s, 5s timeout) to detect stale connections before they cause failures, and a 32MiB client receive cap (raised from grpcio's 4MiB default in #3638, where GetObjectEvents and GetObjectsByState replies outgrew it and failed as RESOURCE_EXHAUSTED). The cap is client-side only; the server sets no send cap, so an oversized reply is still serialised before it fails, which is why the durable fix is bounding replies rather than raising the cap again -- see PLAN-grpc-bounded-replies.md. The database gRPC server uses a 64-thread pool to handle concurrent requests from all daemons. The database client in mariadb.py (_grpc_call) retries UNAVAILABLE and DEADLINE_EXCEEDED failures, rebuilding the channel on a wedged subchannel but keeping it on a refused connection so round_robin can serve the retry from a surviving gateway. All gRPC failures are logged at ERROR level.

get_objects_by_state() returns None on non-retryable errors (distinct from [] for no matches). All object iterators handle this by falling back to unfiltered scans, ensuring that such failures do not silently drop objects from iteration results (e.g. interfaces during instance deletion). A database outage instead raises DatabaseUnavailable out of the iterator.

Object iterators read from MariaDB via get_all_*() functions. All object static values now live in MariaDB.

Cluster Operation Tracking

Every operation schema's model class declares its target objects via a target_fields: ClassVar[dict[str, ObjectType]] class variable. When enqueue_cluster_operation (in schema/operations/util.py) writes the cluster_operations row, it reads that declaration and writes one cluster_operation_targets row per non-None target field. Callers have no per-target bookkeeping obligation — targets are recorded automatically.

The setter _set_last_cluster_operation is private and used only by the internal enqueue plumbing. It is not a public API and should not be called directly.

DatabaseBackedObjectWithOperations exposes two read shapes over the history:

  • last_cluster_operation (property): returns the most recent target row regardless of its state. Consumed by external_view() projections and runs_after=[...] chains, which want the latest pointer independent of whether it has reached a terminal state.
  • has_pending_cluster_operation() (method): returns True if any target row's operation is in {queued, preflight, executing}. Consumed by Network.is_okay() and any future history-aware gate. The query joins cluster_operation_targets against object_states, so a later terminal operation cannot mask an earlier in-flight one.

Because the table is append-only it is bounded by a periodic prune in the cluster daemon, alongside the existing delete_stale_transfers cleanup. The prune removes rows older than CLUSTER_OPERATION_TARGET_RETENTION seconds whose operation has already reached a terminal state. In-flight operations (queued/preflight/executing) are never pruned regardless of age. Because the cluster daemon already runs cluster-wide cleanup under ClusterLock election, no additional locking or master-node gating is required.

Cluster Operation Storage and Work Queues

Cluster operation headers and the per-node work queues both live in MariaDB. The cluster_operations table stores the full operation metadata as JSON in metadata_json, with node_uuid, instance_uuid, network_uuid and priority extracted into indexed columns for dispatch-time filtering. The work_queue table stores one row per queued job with claim fields (claimed_at, claimed_by, attempts) on the same row -- MariaDB row locking replaces the old etcd two-prefix (/sf/queue/, /sf/processing/) design. Dequeue uses SELECT ... FOR UPDATE SKIP LOCKED so concurrent workers either claim distinct rows or one gets nothing.

Creating a cluster operation is atomic: the CreateAndEnqueueCluster gRPC RPC writes the cluster_operations row, the object_states row, and the work_queue row in a single MariaDB transaction. Audit events are written directly into MariaDB via the local spool drainer's mariadb.record_event_batch call.

The cluster daemon runs reap_stuck_cluster_operation_jobs() from daemons/cluster/scheduled_tasks.py on a one-minute schedule. For every row whose claimed_at is older than CLUSTER_OP_STUCK_THRESHOLD seconds, the reaper either clears the claim so a fresh worker picks the job up or -- if attempts has reached CLUSTER_OP_MAX_ATTEMPTS -- deletes the row and transitions the underlying cluster operation to STATE_ERROR. Reaper activity is exported on cluster_op_reaper_requeued_total and cluster_op_reaper_rejected_total, scraped from CLUSTER_METRICS_PORT on the cluster daemon.

The cluster daemon also runs two other reaping sweeps from daemons/cluster/scheduled_tasks.py. per_deleted_object_checks() (every 15 minutes) hard deletes objects that have been in a final state (deleted, complete, abort) for longer than their grace period; its work queue holds (object_type, uuid) tuples fetched with the age filter pushed down to SQL, and objects are hydrated one at a time at processing time inside a per-item exception guard. reconcile_orphaned_objects() (hourly) removes "phantom" object_states rows whose static-values row is gone (with an age guard so mid-creation objects are never raced) and repairs "zombie" static rows that have no state row by writing a deleted state row once the zombie has been seen on two consecutive sweeps; node and namespace objects are excluded from zombie repair. Both kinds of orphan are otherwise invisible to every state-driven iterator.

The elected cluster node also runs reconcile_scheduler_capacity() every five minutes. That cadence is anchored to a cluster-wide last-run stamp (the SCHEDULED_TASK_LAST_RUN_RECONCILE_SCHEDULER_CAPACITY key in cluster_config) rather than to process start, and the elected maintainer's own maintenance pass forces a reconcile as soon as it sees an active hypervisor with fresh metrics and no scheduler_node_capacity row, instead of waiting out the cadence. Both exist because the reconciler is the only thing which creates capacity rows, and a node without a row is admitted against nothing at all: a process-local five minute timer left every placement in a new cluster's first minutes unguarded, and the first pass then recorded the resulting over-limit usage on the row it created (issue 4087). The check used to be a one-shot on the election path, which is why it missed a cold cluster entirely -- election happens seconds after start-up, before any hypervisor has published, so the pass it forced had nothing to size. See the subsystem internals for the predicate it applies now. One pass is a single ReconcileSchedulerCapacity RPC which expires stale namespace claims, re-derives per-hypervisor limits from the typed node_metrics columns, recomputes usage counters from placed instances and the decaying expected-demand signal, and rebuilds the cluster_capacity singleton. The reconciler recomputes the three capacity tables (scheduler_node_capacity, namespace_claims, cluster_capacity) wholesale; the atomic admission and release RPCs also write them incrementally, and are the sole drawdown path against them, so a divergence between what the reconciler computes and what the counters hold is drift, healed on the next pass rather than expected steady state. The claim CRUD RPCs are a third writer: they move capacity between namespace_claims and cluster_capacity (a claim's limits into claimed_*, its namespace's existing drawdown out of unclaimed_used_* and onto the claim, and the reverse on deletion) but consume none, so the same drift-healing property holds. See subsystem internals. Observability is the scheduler_capacity_* gauges and counters exported on CLUSTER_METRICS_PORT, plus one structured log line per pass.

Batched, Priority-Aware Dequeue

sf-net and the sf-queues worker pool both call a single mariadb.dequeue_work_items(queue_names, limit) primitive. The caller passes the queue names in priority order (index 0 = top priority); MariaDB honours that order via ORDER BY FIELD(queue_name, ...), scheduled_at so one SELECT returns the most important eligible work first across an arbitrary number of queues. The previous one-RPC-per-queue polling loop is replaced by one RPC per iteration regardless of how many priority lanes the worker drains.

Lower-priority rows only spill in when the higher-priority queues yield fewer rows than limit, so sustained heavy load on user_facing can still crowd background out -- explicit fairness (bounded staleness or reserved-slot) is intentionally deferred to a follow-up step. Worker crash recovery is unchanged: any claimed- but-not-yet-executed rows that the worker doesn't run are picked up by the stuck-row reaper described above.

To see what that costs an operation in practice, tools/queue-wait-report.py reports the queue-wait distribution by queue class, priority lane and operation type, and tools/operation-timeline.py splits one operation's wait into the initial queue sit, the deferral the dispatcher asked for and the sit after each redelivery -- which is what distinguishes a lane that is genuinely backed up from one whose operations are waiting on a dependency. Both are documented, with their caveats, under networking observability, where they sit next to the other queue tooling.

Coalescible Operations

Some operation tasks are idempotent reconciliation work whose effect depends only on current DB state, not on the count of pending ops asking for it. The canonical example is network_apply_update_dnsmasq: six instance starts on the same network each enqueue one, but the resulting dnsmasq config covers every lease no matter whether the worker ran it once or six times.

Op classes that have such tasks declare them on the class:

class NetOp(BaseClusterOperation):
    coalescible_tasks = schema.COALESCIBLE_TASKS
    coalescible_key_columns = ('network_uuid', 'node_uuid')

coalescible_key_columns names a tuple of indexed columns on cluster_operations, not a single one: a sibling has to match every column before it counts as the same work, so the key can be narrower than "the same network" without any query rewriting. A column with no value on the operation binds IS NULL rather than being dropped from the comparison -- that distinction matters here. NetOp's two cluster-wide tasks (network_apply_update_dnsmasq, network_apply_create_network_node) carry node_uuid = None, so they fold only each other; network_ensure_mesh, enqueued per participating hypervisor, carries its own node's uuid and folds only that node's own siblings. Both are strictly narrower than the network-alone key this replaced, which is the property the whole widening rests on. Binding None as IS NULL was a correction made mid-phase: an earlier version of the preflight refused a None key value outright, which would have silently switched off the only coalescing the cluster does the moment the key grew a second column -- see decision 8 in docs/plans/PLAN-queue-performance-phase-11-multi-column-key.md.

The fold runs at two layers, both controlled by this metadata:

  • Enqueue-side dedup (mariadb.find_existing_coalescible_op): create_and_enqueue in the schema module checks for an existing pending single-task coalescible op matching every key column before inserting a new row. If found, the new caller's op_uuid is the existing op's op_uuid. All raise_for_error waiters then block on the same op and the worker runs it once.

  • Worker-side fold (mariadb.claim_coalescible_siblings): inside BaseClusterOperation.execute, the survivor atomically transitions every other pending coalescible op matching the same key to STATE_COMPLETE in one SQL statement. When the dispatcher surfaces a folded sibling's work_queue row, the terminal-state branch drops it cleanly. A 'coalesced sibling ops' event on the survivor records the folded uuids.

The enqueue-side dedup is the cheaper of the two -- the row never gets inserted -- but the worker-side fold is the safety net for the race where two concurrent callers both lose the lookup.

The enqueue-side dedup will not adopt a slower lane

Reuse is one-sided. create_and_enqueue passes find_existing_coalescible_op the list of priorities it is willing to adopt, which is every PRIORITY at least as urgent as its own. Adopting a more urgent pending op costs the caller nothing: the work runs sooner than it asked for. Adopting a less urgent one means adopting its queue -- the name is {target}-{family}-{priority} -- so the caller sitting in raise_for_error(), and any runs_after dependency hanging off the returned op, wait out the slower lane's queue-sit tail instead of their own.

That overlap is not hypothetical for either coalescible task with more than one enqueue site. Network.ensure_mesh enqueues network_ensure_mesh at user_facing while the network maintainer enqueues the same task, for the same network and node, at background -- and it does so precisely when is_mesh_okay() reports drift, which is the state a network is in immediately after an instance starts elsewhere on it. create_on_network_node and the maintainer straddle user_facing/background for network_apply_create_network_node in the same way, and have done since before the mesh task became coalescible.

The worker-side fold needs no equivalent rule. It runs at dispatch, on an operation which has already been dequeued, so a survivor at any priority is about to do the work immediately; the queue a folded sibling was sitting on no longer matters.

Both guards are key-aware and family-aware

cluster_operations has no queue column, so the key is the only thing either primitive's SQL can use to confine a fold to the operations one dispatcher process actually drains. But naming node_uuid in the key is necessary, not sufficient: which dispatcher drains a per-node operation also depends on the queue family the enqueue chose, because enqueue_cluster_operation builds the queue name as {target}-{family}-{priority}. A node target only reaches sf-net's per-node {node}-network-* queue when the caller also passes family='network'; the default clusteroperation family routes the same target to sf-queues instead, which fills spare worker slots from a single unpartitioned dequeue call and has no per-target routing at all. sf-net, by contrast, hashes every operation for a given target to one worker thread (_routing_key in shakenfist/daemons/network/workitem.py), so a fold there can never mark an operation complete while another thread is executing it. That guarantee is the load-bearing part of the safety argument for a per-node fold, and it does not extend to sf-queues -- see the PARTITIONED-WORKER SAFETY INVARIANT comment at the top of Job.__init__ in that file for the argument in full, including why NodeNetOp.network_apply_create_hypervisor (whose model already carries both columns) stays out of COALESCIBLE_TASKS.

Both guards therefore test the family as well as the key:

  • The enqueue-time guard (InvalidCoalescibleEnqueue in shakenfist/schema/operations/net_op.py) refuses a coalescible task enqueued to a non-networknode target unless node_uuid is in coalescible_key_columns, the operation's derived node_uuid is set, and the enqueue is on the network family.
  • The fold-time guard (BaseClusterOperation.execute) admits a per-node queue on the same condition, read off the queue name's {target}-{family}- prefix, and records the outcome key_cannot_distinguish_queue when it refuses. That name replaces not_cluster_wide; tools/queue-wait-report.py still maps the old name out of retained log history rather than dropping it, since it was the recorded outcome for every build before the key became multi-column.

Two further outcomes exist so that a fold which did not happen is never reported as one which happened and matched nothing -- the reading that let #3878 sit undetected for three months:

  • key_column_missing: a key column which is not an attribute of the operation class at all. That is a mis-declaration, and it is a different thing from a column which is present and None. None binds IS NULL, which is how a cluster-wide NetOp folds only the other cluster-wide operations on its network. A column the class does not have is NULL on every row too, so binding IS NULL for it would quietly degenerate the key to whichever columns did resolve -- for a NetOp, back to the network alone, which is the cross-node fold this design exists to prevent. The fold therefore tests hasattr and skips, rather than reading the value with a default.
  • coalescing_unavailable: the database service answered UNIMPLEMENTED, meaning a rolling upgrade against an sf-database predating the V2 RPCs. The wrappers raise exceptions.CoalescingUnavailable for this rather than returning the same "nothing matched" they return on every other failure, so an operator can watch the upgrade window open and close in the same table as everything else instead of grepping daemon logs.

The V2 gRPC methods

ClaimCoalescibleSiblingsV2 and FindExistingCoalescibleOpV2 take a repeated CoalescibleKeyPair keys in place of the V1 RPCs' scalar target_column/target_uuid, as new RPC names rather than a new field on the existing messages. A repeated field would be wire-compatible but unsafe across a rolling upgrade: an old sf-database would silently ignore the extra columns and fold on the network alone, which is exactly the cross-node corruption the wider key exists to prevent. An old server answers a V2 call with UNIMPLEMENTED, and the client treats that as "coalescing unavailable" and skips the fold rather than falling back to V1 -- a loud, temporary loss of an optimisation instead of a quiet correctness failure. "Loud" means the coalescing_unavailable outcome above, not merely a log line. The V1 RPCs are unchanged and stay for one release.

FindExistingCoalescibleOpV2 additionally carries repeated string priorities, the priority filter described above. Unlike a key column, an old V2 server ignoring that field is only a lost optimisation -- it deduplicates the way the code did before the field existed -- so it travels as a field rather than needing a V3.

📝 Report an issue with this page