Skip to content

Coding rules learned the hard way

Each rule below came out of a real defect. They are not style preferences: breaking one has previously shipped a bug. Read this before touching authorisation predicates, parsers, lookup keys, or metrics.

Never restate a visibility predicate

A listing endpoint filters; a single-object endpoint cannot, because it has already resolved the object by the time it knows who is asking. The two must still agree, so the single-object guard calls the same function the listing filters with rather than open-coding the equivalent test.

For artifacts that function is namespace_or_shared_filter(namespace, obj): own namespace, a namespace whose trust list names the caller, system, or shared. requires_artifact_access calls it.

The ownership predicate has a function too, and it is api_base.namespace_owns(obj_namespace, caller_namespace): the caller's own namespace, or system. requires_artifact_ownership is the deliberately stricter mutation guard and calls it, as do requires_instance_ownership, requires_network_ownership, requires_operation_ownership, the label and artifact write routes, and safe_get_network_interface. It consults neither the shared flag nor the trust list: sharing publishes an object for reading, and a trust is a visibility grant, so neither one hands out a delete button. Creating an object in a namespace that trusts you is a separate question and is still allowed.

Access and ownership are two predicates and not one. Do not reach for namespace_or_shared_filter on a route which changes the object, and do not reach for namespace_owns on a read route — that is the inversion this rule records, in each direction.

Neither predicate is enforced by memory. The unit test test_namespace_ownership_predicate.py parses every file in the package and fails the build if the body of namespace_owns — any membership test against a literal naming 'system' — reappears outside the helper itself. That guard is what stops the restatement count climbing past the twelve sites #3640 and

4887 converted; the predicate had been open-coded at all twelve, across

instances, networks, artifacts, labels, agent operations and network create.

This is not a style preference. requires_artifact_access used to restate the rule as if a.shared and requestor not in [a.namespace, 'system'], which is inverted in both directions, and because arg_is_artifact_ref resolves a UUID straight to Artifact.from_db with no namespace filter, that decorator was the only guard on the path. Any caller who knew a UUID could read any namespace's unshared artifacts. Refuse with 404 rather than 403 so the refusal does not confirm the object exists, and pair every new refusal test with a control that shows the same request succeeding when the one thing under test changes.

The same predicate governs how a name resolves on the read routes, not just whether a resolved object is allowed through. arg_is_visible_artifact_ref (paired with requires_artifact_access) resolves through Artifact.from_db_by_ref_visible_to, which searches the caller's own namespace first and only then widens to what namespace_or_shared_filter admits. Two rules to preserve if you touch this:

  • The caller's own namespace must win, or sharing an object silently retargets every tenant who already used that name.
  • Routes which change an object use plain arg_is_artifact_ref and resolve names narrowly. The ownership guard already refuses the write whichever way the object was named, so this is defence in depth rather than the only gate — but a name must never resolve into someone else's namespace on a route that then deletes what it found. New route, ownership guard, narrow ref decorator — the pairing goes together.

The same split applies to url resolution, and this is where it was missed. Artifact.from_url filters by namespace_or_shared_filter, so it can return an artifact belonging to whoever shares with or trusts the caller. That is the right answer for a caller which will read the result and the wrong one for a caller which will write to it: the upload and cache routes used it to pick a write target, so a trusted namespace could name the owner's source_url and have its own blob added as the newest version — and because add_index ends in delete_old_versions, the owner's older versions went with it. Write paths use Artifact.owned_from_url(), which resolves by ownership and does not create.

Two rules fell out of fixing it, and both generalise past artifacts:

  • A predicate is part of a function's contract, not an implementation detail. If one lookup serves both intents, say which it is in the name and make the other one a separate function. A default that silently suits readers is how a write path inherits a read's authorisation.
  • Creating and modifying are different grants. A trust may let a namespace give you an object it did not have — additive, and the operator guide promises it. It must not let that namespace replace what an object you already own resolves to. When a route can do either, authorise the two cases separately rather than once at the top, and put the audit event after the check so a refused caller cannot write to the event log of a namespace it is about to be told does not exist.

The remaining call sites were narrowed in #3640, and one of them was worse than the sweep recorded. LabelEndpoint.post had been read as safe because it builds its URL from the request — but _label_url accepts <namespace>/<label> and hands back the namespace it was given, so any authenticated caller could push a version into any namespace's label. The requires_admin=True in its swag_from is documentation and enforces nothing, and the route carried no ownership decorator, so nothing stopped it. Two lessons worth carrying:

  • "Built from the request" is not the same as "not caller-controlled." The URL was assembled by our code out of a value the caller chose. Follow the value, not the construction.
  • A read path can be broken in a way that hides the write path's bug. LabelEndpoint.get and delete had answered 500 to every request since 2024 (a pair handed to a filter expecting a string, and a 404 that was computed but never returned), so nobody exercised the endpoint hard enough to notice what post would accept.

The instance path is the case where the obvious narrowing was the wrong fix, and it is worth knowing why before someone "simplifies" it:

  • Resolving disk.base by ownership alone would have broken sharing. Reuse is the entire point of a shared image, and transfer_image treats an artifact with no versions as "cluster does not have a copy", so giving every namespace its own artifact would have meant its own download and its own stored copy of every shared image.
  • The split is therefore per verb, not per artifact. A visible foreign artifact is resolved to a blob and booted from — the same move the label and snapshot branches already made — and never fetched into. owned_from_url() picks the write target; from_url() still picks what you may read.

Artifact.owned_from_url_or_new() exists for the write paths whose target namespace is fixed as the caller's own, or already authorised: they have no two cases to tell apart, so they get the create for free. Routes which accept a caller-nominated namespace must still authorise creating and modifying by hand, which is why owned_from_url() itself does not create. The artifact fetch and upload routes both spell the two cases out for that reason, and the apparent duplication between them is the authorisation rather than a missing abstraction.

One more lesson, from the review of that change:

  • A sweep is only as wide as the directory it was run over. The original #3640 audit listed three sites because it looked at external_api/ and operations/. Instance.snapshot() is a fourth, in the core object, and neither the issue nor the first draft of the fix saw it. It was not exploitable — the URL carries the instance UUID and type_filter pins the type — but "not reachable today" is an argument for the guard being cheap, not for going without it. Grep for the sink (add_index), not for the callers of the resolver you happen to be changing.

Credential-carrying routes are not logged, not redacted

Two independent loggers see a request body: app.py's before_request audit event, and log_request in base.py, which merges the parsed body into the decorated method's kwargs and logs those. Both ask api_base.handles_credentials() — one predicate, in one place — and drop the whole body when it answers yes.

Redacting by field name was tried and is wrong. key means a metadata key name on most endpoints and a secret on a few, so a name-based rule has to know which route it is on anyway, and it silently starts leaking the day somebody adds a route it has not heard of. That is exactly what happened: the federated exchange's credential field is token, which was on neither redaction list, so identity tokens were logged verbatim at INFO and shipped to the log aggregator.

Every route which takes a credential lives under /auth/. Keep it that way, or extend the predicate — never the redaction lists.

A check that runs after the parse is not a check

The endpoint-method decorators are not the outermost thing in a request. log_request calls get_json(force=True) before any method body runs, so a size or shape check written inside a post() cannot prevent work that has already happened. Anything protecting an unauthenticated endpoint from attacker-controlled input has to be an @app.before_request hook registered ahead of log_request_info — see limit_federated_body_size.

While you are there: flask.request.content_length is None for chunked transfer encoding. Treating unknown as small enough lets any caller opt out of a size limit by choosing a header, so refuse with 411 rather than measuring.

Two records must not claim one lookup key

federation.issuer_for_token resolves a trusted issuer by scanning for a matching issuer_url. Uniqueness on that column is therefore a correctness property, not tidiness: two live records claiming one URL make which provider's signing keys are trusted depend on listing order, so an administrator repointing an issuer would believe they had while some requests kept verifying against the old JWKS. The create and update endpoints refuse a duplicate by calling federation.issuer_claiming_url, the same function the resolution path uses.

A check-then-write is not an invariant on its own. issuer_url lives in the attributes row and has no unique index -- and cannot easily get one, because a soft-deleted issuer keeps its row and its URL is deliberately reusable -- so both endpoints hold _issuer_url_lock() across the check and the write it guards. Without that, two administrators configuring one provider at the same moment both read "free" and both write. This is the vsock_cids pattern from instance.py: where a unique index cannot be the arbiter, a cluster lock around the check-then-act has to be.

Put the meter above the expensive thing, not below it

/auth/federated is unauthenticated, so every step above the rate limit is a step an anonymous caller gets for free, as often as they can send. Ordering there is a security property, and the question to ask of each step is not "is this cheap?" but "does this touch the database or the network?". Issuer resolution reads once per configured issuer, so it belongs below the meter even though there are only ever a handful of issuers; only the argument checks, which touch nothing but the request, belong above it. The original ordering had this backwards on its own stated logic -- it placed the meter below the lookup to avoid writing a counter row, when the lookup it was skipping cost more than the row did. If you add a step to that endpoint, place it relative to enforce_rate_limit on that basis, and say why in a comment.

A guard has to sit where the exception is raised

CorruptMappingRule comes from decoding bound_claims or scopes, which only happens on the attributes read. MappingRule.from_db_by_name reads the static row and the object state, so wrapping the lookup in a try looks like protection and is none. Before writing a guard, find the raise site; before believing a regression test, check that the thing it patches is on the path that can actually fail. Use MappingRule.policy() to read the whole policy in one go when you need more than one field -- it is also the single place to catch this.

The same guard has to answer differently in different places, and that is not inconsistency. The exchange endpoint refuses a damaged rule, because bound claims it cannot read are bound claims it cannot check and minting anyway would be authorising on a guess. MappingRule.external_view() describes one, with an explicit unusable marker, because the CRUD routes exist to tell an owner which rule is broken -- raising there turned one bad row into a 500 that hid every healthy rule in the namespace, and made a successful delete report failure on the one call that would have cleaned it up. Ask what the caller will do with the answer before choosing.

A cleanup step must not be abortable by its own instrumentation

psutil.Process.wait() documents NoSuchProcess and TimeoutExpired, and ManagedExecutable._wait_for_exit() caught both. On Linux it waits with pidfd_open() and falls back to its own waitpid() loop for exactly four errnos, re-raising anything else as a bare OSError. A process which has been SIGKILLed but not yet reaped can still be found by pid while no longer having a thread group leader, and pidfd_open() answers EINVAL in that window rather than ESRCH -- so roughly once in 1,600 kills on sfcbr, waiting for a dying dnsmasq threw OSError out of ManagedExecutable.terminate(). The configuration directory terminate() removes next was left behind, and the network operation worker caught it as "Ignored error in net_op" and skipped the rest of that network's node teardown (#4753).

Two lessons, and the second is the general one. Catch what a library's implementation can raise, not only what its documentation names. And a step added to make a teardown safer -- this wait exists because dnsmasq's replacement raced the dying process's sockets (#4472) -- must not be able to abort the teardown it was added to. Observing an exit is instrumentation; when instrumentation cannot answer, it says so and the cleanup still runs.

Releasing a lock is not part of the work

NodeLock.__exit__ was the whole unlock request and nothing else, so a nodelock daemon which went away between the acquire and the release turned a completed operation into a failed one. On sfcbr during an ordinary rolling restart, _apply_delete_on_hypervisor ran its whole teardown, left the with block, and then raised ConnectionResetError out of the unlock: the instance was gone, the network teardown had run, and the delete operation was marked failed anyway, so the client retried into an operation which found everything already deleted (#4902). Three milliseconds later the enclosing instance lock's release failed the same way.

A context manager's __exit__ has a second, worse failure mode: an exception raised there replaces whatever the body raised. The real error becomes a message about tidying up after it.

So a release failure is logged, not raised — ClusterLock.__exit__ has swallowed its own LockNotHeld for this reason far longer. For a node lock that is also safe, and the reason is worth stating because it is a property of the daemon rather than of the lock: daemons/nodelock/main.py holds its lock table as a plain dict local to main() and deletes a key before it answers, so a daemon which cannot answer an unlock has either dropped every lock it held or has already released this one. There is nothing leaked to clean up.

The exception is a request which never arrived, because a nodelock has no lease and so really would stay held until the daemon restarted — and a refused connection is not proof the daemon is gone, since a healthy daemon with a full listen backlog refuses connections too. __exit__ therefore retries the unlock NODELOCK_UNLOCK_ATTEMPTS times before it gives up and logs.

UnknownNodeLockReplyException is deliberately outside NODELOCK_TRANSPORT_ERRORS, the one list of what a restarting nodelock daemon looks like to a client: a reply which parses but carries no known field is a bug in this protocol, not a daemon blip, and it stays loud in __exit__ and in health_check_nodelock() alike.

Fail closed on a field, not on a formatting accident

A reply that says "this went wrong" must say so in a field set only on the success path. CountFederatedAttemptReply used to signal failure by carrying a non-empty error, and the client read anything else as an answer -- so an exception raised with no args, whose str() is empty, arrived as attempts=0, error='' and was read as "nobody has tried this minute, allow". That is the one direction a rate limiter must never fail, on the one endpoint anybody may call.

Both federation replies now carry bool ok, set only where the work actually succeeded, and the client tests that. error is diagnostics. When you add an RPC whose reply has a permissive-looking default -- a zero count, an empty list, a False that means "go ahead" -- carry an explicit success field rather than inferring one, and write the test that returns the empty-error reply and asserts the refusal. The invariant is not that today's code produces a message; it is that no reply can be mistaken for a permissive one.

A state an object can leave is not a terminal state

A guard which asks "is this object gone?" must answer yes only for a state the object cannot leave. The namespace delete handler used to count an instance in error as already gone, alongside deleted. But error is where an instance whose delete failed ends up, and that instance still holds its disks, its interfaces and its placement. Its row is reachable only through its namespace, so deleting the namespace over the top of it left a row with no owner and nothing which would ever sweep it (issue 4847).

Worse, the refusal expired into permission. A failing delete walks delete-wait -> delete-wait-error -> error, and only the middle state blocked, so a caller who retried the namespace delete a moment later got through over exactly the instance the first refusal had protected.

When a guard lists the states it lets through, check each one against the state machine: can an object leave this state, and does it still hold anything while it is in it? deleted is the only state which answers no to both. Test the guard against every state on the failing path, not just the one the guard was written for: each of those is a moment a caller can retry. See the comment in AuthNamespaceEndpoint.delete() in shakenfist/external_api/auth.py.

or [] is a decision about what a failed read means

mariadb.get_objects_by_state() returns None when the read failed and [] when nothing matched, and says so in its docstring. Every or [] at a call site erases that distinction, and the erasure is not neutral: it asserts that the caller treats "we could not find out" and "there is nothing" the same way. Sometimes that is true. Decide it deliberately, because for one caller in this codebase it was catastrophically false.

get_active_blob_uuids() ended in or []. The cleaner uses its result as a complement set -- it unlinks every blob file on disk whose uuid is not in the list -- so a failed read arrived as "no blobs are active", which is an instruction to delete the node's entire blob store. The trigger was not exotic: an oversized GetObjectsByState reply (issue 3638) is a non-retryable RESOURCE_EXHAUSTED, raised as-is by _grpc_call, mapped to None by the client wrapper, and flattened to [] one line later. The same call in the cluster daemon only ever iterates the list, so there the empty list was merely a skipped pass. One accessor, two callers, opposite consequences.

So the rule is about the caller, not the accessor. Before collapsing an error into a value, ask what each caller does with it. A caller that iterates can usually tolerate a skipped pass -- catch explicitly, log, and move on. A caller that complements, gates, or diffs against the list cannot, and must see the failure. get_active_blob_uuids() now raises exceptions.DatabaseUnavailable, which is the same shape as the fix for issue 3373 (an unreachable database must not be indistinguishable from a missing object) and gets the REST path a clean 503 for free via handle_database_unavailable.

A sweep that reads a work list has a quieter version of the same problem: it does not delete anything wrongly, it just does nothing, reports a healthy pass over an empty queue, and lets the backlog grow -- which in issue 3638 grew the very reply that could not be read. _sweep_work_list() in the cluster daemon's scheduled tasks is the shared answer: a failed read returns None, is counted into cluster_sweep_work_list_failure_streak, and is logged as a skipped pass. The general form: silence is not success, and an empty result set is not the absence of an answer.

One trap when handling this deliberately: a failed read from the mariadb client arrives in two shapes. The client wrappers map grpc.RpcError to a None/False/[] return, but _grpc_call() raises exceptions.DatabaseUnavailable once its retry budget is spent, and that is deliberately not an RpcError subclass (issue 3373), so it propagates through the wrapper untouched. Handling only the return value covers the oversized-reply case and misses the tier outage -- which is the more likely reason a read fails, and the condition an alert on the streak most needs to see. Cover both, and prove it with a test that sets side_effect = DatabaseUnavailable(...) rather than a None return. Code which is meant to ride out an unreachable database catches DatabaseUnavailable explicitly and says so; Daemon.check_daemon_state() and ClusterLock.__enter__ are the examples to copy.

Having covered both, do not then treat them alike inside a loop. The two shapes differ in blast radius. A None return is per-reply, so the next object type is still worth reading. A DatabaseUnavailable is tier-wide: every remaining read will fail identically, and each will spend a full _grpc_call retry budget before it does. Continuing therefore costs one budget per iteration, and both the deleted-object sweep (28 object types) and orphan reconciliation run inside a single scheduled job whose watchdog is only petted between jobs. The tolerant continue that is right for the per-reply case would turn a database outage into a SIGABRT of the elected cluster maintainer and a lock failover. Continue on the per-reply shape; stop the loop on the tier-wide one. The general form: when one read becomes N reads, recompute the caller's worst-case wall time rather than inheriting it.

And when you stop a loop early, ask what it does to whatever is behind the stop. Both of those loops iterate a fixed list of object types from the start of the list every pass, so break on its own would mean a single persistently slow type hides every type after it forever -- and the backlog that then accumulates makes that type slower still, which is the #3638 ratchet rebuilt one level up. They keep a resume offset and begin the next pass after whichever type stopped the last one, so the pass stays bounded and every type is reached within a bounded number of passes.

The same rule applies to the reply itself, one layer down. A gRPC reply whose only payload is a repeated field has nowhere to put "the read failed", so returning object_uuids=uuids or [] from a servicer hands the client the same ambiguity the or [] at a call site does -- except the client cannot even see that a decision was made. DatabaseServicer.GetObjectsByState did exactly this, which left the blob-store deletion hazard fully reachable through the likelier failure (MariaDB down while sf-database is up and answering) after the client-side half had been fixed. Signal it on the status instead: context.set_code(grpc.StatusCode.UNAVAILABLE) for a transient database error, which _grpc_call retries and converts to DatabaseUnavailable, and INTERNAL for a handler bug, which is non-retryable and so becomes a None return. When you fix an empty-means-failure bug, check both ends of the wire.

And check every operand of the decision, not just the one that led you there. The cleaner's test is an or:

if (blob_uuid not in active_blob_uuids
        or blob_uuid not in all_node_blobs):
    os.unlink(entpath)

Hardening get_active_blob_uuids() fixed the first operand and left the store exactly as deletable through the second, which reads the node's own blob locations and flattened failure to [] at all three layers. Worse, the second operand fails more often than the first, because a plain MariaDB OperationalError -- a lock wait timeout, a deadlock, a dropped connection -- breaks it while sf-database itself stays healthy and answers everything else normally. A guard on one input of a multi-input decision is not a guard on the decision. Enumerate the inputs, and write the negative-control test for each.

That said, the fix is per-caller and not per-accessor, so it does not follow that every accessor should raise. get_references_from() still collapses a failed read to [], deliberately and with the reasoning in its docstring, because every one of its callers iterates. The complement-set caller gets its own raising accessor, get_node_blob_uuids(), built on the truthful internal _get_references_from(). Two accessors over one read, named for what their callers may assume, beats one accessor with a raise_on_failure flag that every call site has to remember to set.

Cluster CI tests only run in the merge queue

The functional cluster lanes in Functional tests -- everything under deploy/shakenfist_ci/cluster_ci_tests/ -- are skipped on pull_request and run on merge_group. A green PR therefore says nothing about whether those tests pass, or whether they can even reach their first assertion. Some lanes do not run in the merge queue at all, only weekly: the Ubuntu base image, the slim-tier topology with Kerbside, and a single machine (see Merge lanes and weekly lanes). The Ansible modules and Node lifecycle suites (ansiblemoduletests.sh and nodelifecycletests.sh, shell scripts rather than cluster_ci_tests/ lanes) also run only weekly, so neither a pull request nor a retest in the merge queue exercises them. To run any of those against a branch, dispatch scheduled-tests.yml on it.

cluster_ci_tests/test_federation.py sat through four commits registering a trusted issuer with an http:// jwks_uri while the API had refused non-HTTPS jwks_uri since the object was added. Every test in the class died in setUp on a 400, and nothing said so until the branch entered the queue.

Two habits follow. When you add or change a cluster CI test, run it, or at minimum drive the validator it depends on directly -- these tests import cleanly given pip install shakenfist-client testtools oslo.concurrency prettytable in a scratch venv, so "the client is not installed" is not a reason to skip verification. And when a test needs input that a validator rejects, the validator is usually right: change the test, never carve a loopback or test-only exemption into a security check. If that makes the test unrunnable, make it skip loudly and file the issue -- see #3639 for the JWKS certificate case.

That issue is a worked example of how the second habit tends to resolve. The test needed the cluster to trust a certificate it had minted, and the tempting fix was a test-only exemption in the deploy. What it became instead was FEDERATION_JWKS_CA_BUNDLE -- extra trust anchors for JWKS fetches, which a self hosted Authentik or Keycloak needs anyway -- with CI as its first user. When a test cannot run because a security check is doing its job, the useful question is usually "what would a real operator need here", not "how do we get around this in CI".

Two traps it also left behind, both of which cost a debugging round:

  • ssl.create_default_context(cafile=...) replaces the system trust store rather than adding to it. Build the default context and then call load_verify_locations on it. The wrong spelling passes every test that only checks the new anchor is present.
  • Python 3.13 enables ssl.VERIFY_X509_STRICT by default, so a leaf certificate with no Authority Key Identifier is refused with "Missing Authority Key Identifier". If you generate certificates in a test, give them AKI, SKI and basic constraints -- and be aware the symptom is indistinguishable from the CA not being trusted at all.

If a cluster test can run on one node, share it with the smoke suite

cluster_ci_tests/test_database_tier.py landed with two independent defects at once (#3694, #3708) and blocked the merge queue for four days, because the merge queue was the first place it ever ran. Both defects were reproducible on a single node.

The two suites are disjoint directories -- smoke-ci.conf discovers only smoke_ci_tests/, cluster-ci.conf only cluster_ci_tests/ -- so "add it to smoke" cannot mean moving it without losing the multi-node coverage. Define the test bodies once in a mixin under deploy/shakenfist_ci/ (not in either suite directory, or stestr collects it twice from one run) and subclass the mixin from both suites. database_tier.DatabaseTierTestsMixin is the worked example: the two tests needing one sf-database run in both, and the load-balancer test needing N>=2 stays in the cluster suite where its skip is honest.

Prefer this whenever a cluster test's preconditions are met by the single-node localhost topology. Sometimes that topology is also the better test: the API and sf-database share a machine there, so it is the most exposed to the direct-MariaDB routing regression #3708 was about.

tearDown runs before addCleanup, and the base class already deletes

BaseNamespacedTestCase.tearDown() deletes every instance in the namespace and blocks until they are gone, swallowing ResourceNotFoundException as it goes. testtools runs tearDown() before the addCleanup stack:

ORDER: ['tearDown', 'cleanup_from_test', 'cleanup_from_setUp']

So self.addCleanup(self.test_client.delete_instance, uuid) in a namespaced test can only ever 404, and unlike tearDown's own deletes it is not guarded -- it fails the test after the assertions have passed. This is what kept test_instance_get_fetches_the_attributes_row_once red even once its measurement bug was fixed. Do not register instance cleanups in a namespaced test; the base class already reaps them. Reserve addCleanup for state the base class knows nothing about, such as the host devices in test_stray_vxlan.py or a namespace the test made itself.

The last thing tearDown() does is delete the namespace, so anything reached through a /auth/namespaces/<namespace>/... route -- a key, a claim, a federation mapping rule -- has to be removed before super().tearDown() is called, not from a cleanup. ClaimAPIMixin in test_namespace_claims.py and TestLoki in test_loki.py are both written that way, and both say why in a comment.

That was not true until 2026. _remove_namespace() asked whether a namespace name was in the result of get_namespaces(), which is a list of external_view() dicts, so the answer was always no and no namespaced test deleted its namespace for six years. The rule above was correct advice for the wrong reason, and it will be correct advice for the right one only as long as the comparison stays fixed -- smoke_ci_tests/test_ci_harness.py is what holds it there. When a helper's failure mode is to succeed quietly, the guard has to assert the effect on the cluster rather than that the helper returned.

A signal handler only sets flags

Python runs a signal handler on the main thread, between any two of that thread's bytecodes, so the handler shares every lock the main thread might be holding at that instant -- and a non-reentrant one it needs means waiting for itself forever. Logging takes one (the Loki handler enqueues under the logship spool's lock, and the main thread logs all the time), and so does the database client. sf-api's drain handler logged and deadlocked its worker until gunicorn SIGKILLed it (#3495). Every daemon's exit_gracefully() logged and wrote its state to the database, and sf-database, stopping under a slow MariaDB, never got as far as its drain (#4601). Neither looked wrong.

A handler assigns, writes the abort file with daemon.write_abort_file(), or calls a function marked @signal_safe (shakenfist/util/signals.py); the main thread notices and does the real work. tools/check-signal-handlers.py enforces this in pre-commit for every signal.signal() handler and, through @signal_safe, every helper they call. When it cannot see that a call is safe -- Event.set() on an event only another thread waits on -- end the line with # signal-safe: <why>, and make the why something a reviewer can check.

A column-to-column comparison has no bind processor

SQLAlchemy coerces a bound value through the column type on its way into the statement. It cannot do that for a comparison between two columns, because there is no Python value to coerce. Any join whose two sides store the same logical value in different physical forms therefore matches nothing, silently, and a query that returns no rows is indistinguishable from a feature with no work to do.

Both of object_states' join columns have that problem:

  • object_states.object_uuid is a sa.String object key holding the dashed uuid form (or, for a namespace, its name), while the static tables use sa.Uuid, which is undashed CHAR(32) on MariaDB. Join through _dashed_uuid_expr(), never a bare sa.cast.
  • object_states.object_type is sa.Enum(ObjectType), which SQLAlchemy stores by enum name ('NET_OP'), while the operation and object tables carry the enum value ('net_op'). Bind an ObjectType member; do not compare the two columns.

The second is the subtler one, because binding a plain 'net_op' string does work -- ObjectType subclasses str, so the member and its value hash identically in the Enum type's _valid_lookup -- which makes every other object_states query in mariadb.py correct by accident and makes the broken join look like all of them.

The two coalescing primitives in mariadb.py had both defects in one ON clause, each individually fatal, and shipped that way for three months of green CI (issue #3878). Every test mocked either the primitive or the engine, so nothing ever ran the statement against a table. The operator-facing symptom was documented before the cause was: the coalesced sibling ops event simply never fired. Both queries now share _coalescible_states_join(), and shakenfist/tests/test_mariadb_coalescing.py runs them against a real database.

The general rule: a query which can fail by matching nothing needs a test that proves it matches something. Asserting the statement's shape against a mocked engine cannot do that, and neither can a test that rewrites the query to suit sqlite -- register the missing SQL functions instead, so the statement under test is the statement that ships.

The same rule has a write-side form, and it cost longer to find. A test which mocks the write helper and asserts it was called the right number of times proves nothing about what the database kept. The v1 cluster_operation_targets table declared unique=True on operation_uuid alone, so a multi-target operation's second and third target rows raised IntegrityError, which the writer swallowed as idempotency -- one row stored out of three. The test enqueued a hot plug and asserted three calls with the right arguments, and passed, for months. The symptom surfaced from the network side only: the first declared target is the one that survives, so instance-side gates read correctly while has_pending_cluster_operation() on the network returned False and the network maintainer recreated a bridge an in-flight operation was about to build. A constraint is part of the statement under test; a call count is not a row count. What holds the invariant now is structural rather than a test: every target travels as one targets= list into create_and_enqueue_cluster_operation, written in the same transaction as the operation, and the table's uniqueness is the composite uq_cot_op_target on operation and target.

A uuid column's type is uuid_column_type(), never a bare sa.Uuid()

sa.Uuid() decides how to bind, read and render itself per dialect. SQLAlchemy 2.0 treated MariaDB as having no native UUID type, so every sa.Uuid() column became CHAR(32) holding undashed hex, and that is what every existing Shaken Fist table contains. SQLAlchemy 2.1 made the MariaDB dialect report native UUID support, and a bare sa.Uuid() stopped converting: uuid.UUID binds reached the driver unconverted instead of as the hex those CHAR(32) columns hold, reads came back as plain strings, and newly created columns would have been MariaDB UUID. Nothing raised. The coalescing keys on cluster_operations.network_uuid and node_uuid simply matched no rows, and the SQLAlchemy 2.1 upgrade failed only in the merge queue, as the coalescing cluster tests reporting the #3878 symptom again.

shakenfist.schema.sqlalchemy.uuid_column_type() returns sa.Uuid(native_uuid=False), which keeps the CHAR(32) behaviour on every SQLAlchemy version. Use it for every uuid column, including the SQLNativeUUID marker's (whose name predates this and is misleading: it has never been a native UUID). shakenfist/tests/test_uuid_column_type.py builds every table mariadb.py defines and fails on any sa.Uuid column that could go native, and checks the bind, result and DDL behaviour against the MariaDB dialect itself. The sqlite-backed coalescing tests could not catch this: sqlite has no native UUID, so a bare sa.Uuid() still behaves there exactly as it used to on MariaDB.

A frozen model is not a deep frozen model

mariadb.get_<type>() returns a Pydantic model with model_config = ConfigDict(frozen=True), and that model is the one entry the process-wide static object value cache holds for its whole TTL. Every reader in the process gets the same object. frozen=True stops attribute assignment on it; it does not stop mutation of the contents of a list or dict field. So data.commands.pop(0) succeeds, and it edits what every other reader on the node is about to see.

Two obligations follow, and they are two different obligations:

  • Copy the container you hand out. A _db_get() or _static_values_to_dict() copying list(data.commands) is the containment boundary. Without it, every object built from one cache hit shares one list, so a mutation anywhere is a mutation everywhere until the entry ages out.
  • Do not mutate an element of a container you were handed. The copy above is shallow, which is deliberate -- a deep copy would defend against a mutation no code performs and would cost an allocation per element per read -- so the dicts inside a copied list are still the cache's. Consume a command list by copying it and popping the copy, the way SideChannelExecutorJob.__init__ does; build the dict you intend to edit locally, the way add_result()'s callers do.

SideChannelExecutorJob did the popping (issue #3516, PR #3970). It aliased agentop.commands as its work queue and popped each command as it dispatched, which drained the operation and, through it, the cache. The visible damage was an inverted decision: operation_is_retryable() opens by refusing an empty command list, so a single-command get-file declared itself unretryable the instant its only command went out -- precisely the operation shape the retry path exists for. The second consequence was quieter: NodeAgentopOp._preflight() iterates the same list looking for put-blob commands whose blobs it must make local, and against a drained cache entry it would find none and skip the preparation with nothing logged.

Note what did not catch this. The class is covered by unit tests which build the executor and drive a dispatch, but they hand-set job.commands rather than going through the constructor, so the alias was created in a line no test executed. A regression test for this class of bug has to build the object the way production does and then assert on what a second reader sees.

The static-values models with container fields today are AgentOperationData.commands and InstanceData's disk_spec, video and side_channels; all four are copied at their boundary. A new container field on a cached model joins that list.

A per-item convenience wrapper does not belong in a loop

instance_usage_for_blob_uuid() answers "which instances use this blob?". It does so by calling instance_blob_usage(), which walks every healthy instance in the cluster: one FindInstances, then a block_devices attribute read per instance and a get_references_from per link of every disk's dependency chain. The wrapper is a single line over a cluster-wide scan, and nothing about the call site says so.

Called once per request that is fine. Called once per item rendered it multiplies the whole scan by the length of the response, and the cost is invisible in the handler:

# Wrong: one cluster-wide instance walk per version.
for idx in ev['blobs']:
    ev['blobs'][idx]['instances'] = instance_usage_for_blob_uuid(
        ev['blobs'][idx]['uuid'])

# Right: one walk, indexed per version.
blob_usage = instance_blob_usage()
for idx in ev['blobs']:
    ev['blobs'][idx]['instances'] = blob_usage.get(
        str(ev['blobs'][idx]['uuid']), [])

Issue #3502 found this in the cluster cleanup loop and fixed it there, and left a docstring on instance_blob_usage() saying which form to use. Four API handlers kept the per-blob form anyway -- an artifact GET walked once per version, the artifact and snapshot listings once per artifact and once per version of every snapshot -- and it took issue

3876, filed against an unrelated symptom, to find them. The lesson is

not "read the docstring": it is that a function whose cost is proportional to the size of the cluster must be seen to be, so a per-item wrapper over one needs the bulk form to be the obvious thing to reach for and the loop body to be plainly an index lookup.

The same shape of trap exists wherever a property hides a query. Blob.depends_on and Blob.transcoded each ran their own filtered get_references_from while the caller had already read the same rows unfiltered; that was the other half of #3876.

A no_log default is a cluster-wide redaction

An option declared no_log: true in a role's meta/argument_specs.yml is not a per-task setting. Ansible registers whatever value the argument spec validated as a secret for the rest of the run, and from ansible-core 2.22 every later occurrence of that exact string anywhere in the output -- task names, assert messages, variables which have nothing to do with the option -- comes out as $REDACTED$.

The value validated is usually the role's own default, because an option an operator did not set still gets validated. The node role defaulted system_key, which the kerbside_credentials entry point declares no_log, to "unknown", so a deploy which left it alone -- any deploy reading the key from Vault, for one -- masked an ordinary English word out of every message of the run. Upstream, the shakenfist.kerbside collection declared the whole kerbside_sources list no_log, which registered each source's type: shakenfist and turned a passing task header into TASK [$REDACTED$.kerbside.kerbside : ...]; that failed the Kerbside wiring test on every pull request in this repository until ansible-core was held below 2.22 (issue #4773, kerbside#687). The role fix shipped in shakenfist.kerbside 0.9.0, and the sanity job now pins ansible-core and ansible-lint exactly, so a new release of either arrives as its own Renovate pull request.

So the default of a no_log option must be a string which cannot occur anywhere else: empty, which the masker ignores, or a change-me-<thing> placeholder matching the example inventories. defaults/main.yml has to agree with the spec, since it is the value a run actually validates. test_collection_no_log_defaults.py checks both. Narrow no_log to the fields which are secret rather than declaring a structure that contains one, and remember that no_log on a task is a different thing -- it suppresses that task's own output and registers nothing.

A backfill cannot finish while the old writers are still running

A migration which adds a column, fills it from somewhere else and then advances the schema version has only filled the rows that existed when it ran. On a rolling upgrade the schema moves first and the daemons restart one at a time after it, so for the length of the deploy every new row is written by code which does not know about the column and takes its DEFAULT. Once the version reads the new number the migration never runs again, and those rows keep the default forever.

event_objects.timestamp shipped exactly that way. Its v1 to v2 migration backfilled from events and its backfill was properly restartable, which is what made it look complete; the rows it could not have reached were the ones not yet written. One production deploy left 30,658 rows at timestamp = 0, and because the per-object read orders on that column they sorted as the oldest events those objects had and dropped out of the recent page (issue #4782).

So when a migration changes how rows are written, something which runs after the restarts has to finish it. For event_objects that is the first stage of the daily events prune sweep (_direct_backfill_event_object_timestamps()), which repairs default-valued rows in bounded batches and, once there are none, costs one candidate query which finds nothing: a sweep that finds nothing is the proof the deploy is over. Re-running sf-ctl ensure-mariadb-schema is not an answer on its own, because the version has already advanced and because an operator has to remember to do it.

Two details worth copying. The repair must be first in a sweep with a wall-clock budget, or a cluster whose other stages exhaust the budget never reaches it. And it must not be a single table-wide UPDATE ... JOIN: that next-key locks every row it examines while every node in the cluster is trying to insert events, which is the defect _direct_prune_events_by_type() already carries the measurements for.

A schema backfill runs in bounded batches

The previous rule's last paragraph says a repair must not be a single table-wide UPDATE ... JOIN. Neither may the migration. sf-ctl ensure-mariadb-schema runs from play 5 of the site playbook, before any node restarts onto the new code, so every daemon in the cluster is reading and writing the table a backfill is rewriting. Under REPEATABLE READ one unbounded UPDATE next-key locks every row it examines, in every table it touches, until it commits -- and a migration-time backfill commits once, at the end.

The event_objects.timestamp backfill did that on a production cluster and held 71.9 million row locks for 31 minutes. For all 31 of them every node's event writes failed with MariaDB 1205 Lock wait timeout exceeded (1,454 of them), clients saw 388 RecordEventBatch DEADLINE_EXCEEDED errors, and object hard deletes -- which delete the object's events -- waited on the same locks (issue #4797). Nothing was wrong with the infrastructure and nothing in the migration failed; it simply held its locks for as long as it took.

So a migration which rewrites more rows than fit comfortably in one transaction calls _migration_backfill_batched(), which walks the table in an indexed column's order and runs the backfill once per batch with :cursor_from and :cursor_to bound to that batch's inclusive range. Each batch is its own transaction, so a concurrent writer waits for one batch at worst, and only if its row falls in the range currently held. test_mariadb_migration_backfills.py reads the source of every _ensure_*/_migrate_* function in mariadb.py and fails on an UPDATE which is not bounded that way, because a new migration with this defect would not fail any other test.

Two things to get right when you write one. The walk has to be a cursor, not a repeated "which rows still need work?" query: a candidate query scans past everything already done to find the next batch, which is quadratic in the size of the table and much slower than the statement it replaced. And the backfill statement keeps its own "not done yet" predicate alongside the range, which is what makes a batch idempotent and the whole migration restartable.

A sample taken before the work is not a decision

NodeInstOp._instance_delete() decided whether to tear a network's bridge and vxlan device off the hypervisor from a list of "networks the node's other instances use" that it built before it deleted the instance. Between the sample and the teardown ran the whole of inst.delete() and one update_dnsmasq() enqueue per network, and in that window an interface hot plug or an instance create landing on the node attaches to one of those networks. The sample says nobody else is using it, so a live domain loses its network -- which does not announce itself as a race, it looks like the bridge going missing for no reason (issue 4958).

The shape to copy was already in the same loop: it re-reads each Network from the database and skips one in delete-wait rather than trusting anything sampled earlier. So ask the question at the point of decision, or hold across both the check and the act whatever lock the thing you are racing takes. A sample is only as good as the window after it, and "it is only a few statements later" is a statement about today's code.

Then, because the records are records and not observations, the host gets a veto. util_network.foreign_bridge_members() asks ip which devices are enslaved to br-vxlan-<vxid> and subtracts the ones Shaken Fist put there itself; anything left is a guest tap, which is proof that a domain is attached right now whatever the records say. The stray vxlan reaper (daemons/network/maintain.py) has applied that veto before mutating for longer, and the two share one definition of which devices are ours so they cannot drift apart. A host which cannot be asked counts as in use, because the two errors are not equal: a bridge left behind is reaped by a later maintain pass, while a bridge taken from a live domain is an outage.

📝 Report an issue with this page