Coding rules learned the hard way¶
Each rule below came out of a real defect. They are not style preferences: breaking one has previously shipped a bug. Read this before touching authorisation predicates, parsers, lookup keys, or metrics.
Never restate a visibility predicate¶
A listing endpoint filters; a single-object endpoint cannot, because it has already resolved the object by the time it knows who is asking. The two must still agree, so the single-object guard calls the same function the listing filters with rather than open-coding the equivalent test.
For artifacts that function is namespace_or_shared_filter(namespace, obj):
own namespace, a namespace whose trust list names the caller, system, or
shared. requires_artifact_access calls it.
The ownership predicate has a function too, and it is
api_base.namespace_owns(obj_namespace, caller_namespace): the caller's own
namespace, or system. requires_artifact_ownership is the deliberately
stricter mutation guard and calls it, as do
requires_instance_ownership, requires_network_ownership,
requires_operation_ownership, the label and artifact write routes, and
safe_get_network_interface. It consults neither the shared flag nor the
trust list: sharing publishes an object for reading, and a trust is a
visibility grant, so neither one hands out a delete button. Creating an object
in a namespace that trusts you is a separate question and is still allowed.
Access and ownership are two predicates and not one. Do not reach for
namespace_or_shared_filter on a route which changes the object, and do not
reach for namespace_owns on a read route — that is the inversion this rule
records, in each direction.
Neither predicate is enforced by memory. The unit test
test_namespace_ownership_predicate.py parses every file in the package and
fails the build if the body of namespace_owns — any membership test against
a literal naming 'system' — reappears outside the helper itself. That guard
is what stops the restatement count climbing past the twelve sites #3640 and
4887 converted; the predicate had been open-coded at all twelve, across¶
instances, networks, artifacts, labels, agent operations and network create.
This is not a style preference. requires_artifact_access used to restate the
rule as if a.shared and requestor not in [a.namespace, 'system'], which is
inverted in both directions, and because arg_is_artifact_ref resolves a UUID
straight to Artifact.from_db with no namespace filter, that decorator was the
only guard on the path. Any caller who knew a UUID could read any namespace's
unshared artifacts. Refuse with 404 rather than 403 so the refusal does not
confirm the object exists, and pair every new refusal test with a control that
shows the same request succeeding when the one thing under test changes.
The same predicate governs how a name resolves on the read routes, not just
whether a resolved object is allowed through. arg_is_visible_artifact_ref
(paired with requires_artifact_access) resolves through
Artifact.from_db_by_ref_visible_to, which searches the caller's own
namespace first and only then widens to what namespace_or_shared_filter
admits. Two rules to preserve if you touch this:
- The caller's own namespace must win, or sharing an object silently retargets every tenant who already used that name.
- Routes which change an object use plain
arg_is_artifact_refand resolve names narrowly. The ownership guard already refuses the write whichever way the object was named, so this is defence in depth rather than the only gate — but a name must never resolve into someone else's namespace on a route that then deletes what it found. New route, ownership guard, narrow ref decorator — the pairing goes together.
The same split applies to url resolution, and this is where it was missed.
Artifact.from_url filters by namespace_or_shared_filter, so it can return
an artifact belonging to whoever shares with or trusts the caller. That is the
right answer for a caller which will read the result and the wrong one for a
caller which will write to it: the upload and cache routes used it to pick a
write target, so a trusted namespace could name the owner's source_url and
have its own blob added as the newest version — and because add_index ends in
delete_old_versions, the owner's older versions went with it. Write paths use
Artifact.owned_from_url(), which resolves by ownership and does not create.
Two rules fell out of fixing it, and both generalise past artifacts:
- A predicate is part of a function's contract, not an implementation detail. If one lookup serves both intents, say which it is in the name and make the other one a separate function. A default that silently suits readers is how a write path inherits a read's authorisation.
- Creating and modifying are different grants. A trust may let a namespace give you an object it did not have — additive, and the operator guide promises it. It must not let that namespace replace what an object you already own resolves to. When a route can do either, authorise the two cases separately rather than once at the top, and put the audit event after the check so a refused caller cannot write to the event log of a namespace it is about to be told does not exist.
The remaining call sites were narrowed in #3640, and one of them was
worse than the sweep recorded. LabelEndpoint.post had been read as safe
because it builds its URL from the request — but _label_url accepts
<namespace>/<label> and hands back the namespace it was given, so any
authenticated caller could push a version into any namespace's label. The
requires_admin=True in its swag_from is documentation and enforces
nothing, and the route carried no ownership decorator, so nothing stopped it.
Two lessons worth carrying:
- "Built from the request" is not the same as "not caller-controlled." The URL was assembled by our code out of a value the caller chose. Follow the value, not the construction.
- A read path can be broken in a way that hides the write path's bug.
LabelEndpoint.getanddeletehad answered 500 to every request since 2024 (a pair handed to a filter expecting a string, and a 404 that was computed but never returned), so nobody exercised the endpoint hard enough to notice whatpostwould accept.
The instance path is the case where the obvious narrowing was the wrong fix, and it is worth knowing why before someone "simplifies" it:
- Resolving
disk.baseby ownership alone would have broken sharing. Reuse is the entire point of a shared image, andtransfer_imagetreats an artifact with no versions as "cluster does not have a copy", so giving every namespace its own artifact would have meant its own download and its own stored copy of every shared image. - The split is therefore per verb, not per artifact. A visible foreign
artifact is resolved to a blob and booted from — the same move the label and
snapshot branches already made — and never fetched into.
owned_from_url()picks the write target;from_url()still picks what you may read.
Artifact.owned_from_url_or_new() exists for the write paths whose target
namespace is fixed as the caller's own, or already authorised: they have no
two cases to tell apart, so they get the create for free. Routes which accept
a caller-nominated namespace must still authorise creating and modifying by
hand, which is why owned_from_url() itself does not create. The artifact
fetch and upload routes both spell the two cases out for that reason, and the
apparent duplication between them is the authorisation rather than a missing
abstraction.
One more lesson, from the review of that change:
- A sweep is only as wide as the directory it was run over. The original
#3640 audit listed three sites because it looked at
external_api/andoperations/.Instance.snapshot()is a fourth, in the core object, and neither the issue nor the first draft of the fix saw it. It was not exploitable — the URL carries the instance UUID andtype_filterpins the type — but "not reachable today" is an argument for the guard being cheap, not for going without it. Grep for the sink (add_index), not for the callers of the resolver you happen to be changing.
Credential-carrying routes are not logged, not redacted¶
Two independent loggers see a request body: app.py's before_request audit
event, and log_request in base.py, which merges the parsed body into the
decorated method's kwargs and logs those. Both ask
api_base.handles_credentials() — one predicate, in one place — and drop the
whole body when it answers yes.
Redacting by field name was tried and is wrong. key means a metadata key
name on most endpoints and a secret on a few, so a name-based rule has to know
which route it is on anyway, and it silently starts leaking the day somebody
adds a route it has not heard of. That is exactly what happened: the federated
exchange's credential field is token, which was on neither redaction list,
so identity tokens were logged verbatim at INFO and shipped to the log
aggregator.
Every route which takes a credential lives under /auth/. Keep it that way,
or extend the predicate — never the redaction lists.
A check that runs after the parse is not a check¶
The endpoint-method decorators are not the outermost thing in a request.
log_request calls get_json(force=True) before any method body runs, so a
size or shape check written inside a post() cannot prevent work that has
already happened. Anything protecting an unauthenticated endpoint from
attacker-controlled input has to be an @app.before_request hook registered
ahead of log_request_info — see limit_federated_body_size.
While you are there: flask.request.content_length is None for chunked
transfer encoding. Treating unknown as small enough lets any caller opt out of
a size limit by choosing a header, so refuse with 411 rather than measuring.
Two records must not claim one lookup key¶
federation.issuer_for_token resolves a trusted issuer by scanning for a
matching issuer_url. Uniqueness on that column is therefore a correctness
property, not tidiness: two live records claiming one URL make which
provider's signing keys are trusted depend on listing order, so an
administrator repointing an issuer would believe they had while some requests
kept verifying against the old JWKS. The create and update endpoints refuse a
duplicate by calling federation.issuer_claiming_url, the same function the
resolution path uses.
A check-then-write is not an invariant on its own. issuer_url lives in the
attributes row and has no unique index -- and cannot easily get one, because a
soft-deleted issuer keeps its row and its URL is deliberately reusable -- so
both endpoints hold _issuer_url_lock() across the check and the write it
guards. Without that, two administrators configuring one provider at the same
moment both read "free" and both write. This is the vsock_cids pattern from
instance.py: where a unique index cannot be the arbiter, a cluster lock
around the check-then-act has to be.
Put the meter above the expensive thing, not below it¶
/auth/federated is unauthenticated, so every step above the rate limit is a
step an anonymous caller gets for free, as often as they can send. Ordering
there is a security property, and the question to ask of each step is not "is
this cheap?" but "does this touch the database or the network?". Issuer
resolution reads once per configured issuer, so it belongs below the meter
even though there are only ever a handful of issuers; only the argument
checks, which touch nothing but the request, belong above it. The original
ordering had this backwards on its own stated logic -- it placed the meter
below the lookup to avoid writing a counter row, when the lookup it was
skipping cost more than the row did. If you add a step to that endpoint, place
it relative to enforce_rate_limit on that basis, and say why in a comment.
A guard has to sit where the exception is raised¶
CorruptMappingRule comes from decoding bound_claims or scopes, which
only happens on the attributes read. MappingRule.from_db_by_name reads the
static row and the object state, so wrapping the lookup in a try looks like
protection and is none. Before writing a guard, find the raise site; before
believing a regression test, check that the thing it patches is on the path
that can actually fail. Use MappingRule.policy() to read the whole policy in
one go when you need more than one field -- it is also the single place to
catch this.
The same guard has to answer differently in different places, and that is not
inconsistency. The exchange endpoint refuses a damaged rule, because bound
claims it cannot read are bound claims it cannot check and minting anyway would
be authorising on a guess. MappingRule.external_view() describes one, with
an explicit unusable marker, because the CRUD routes exist to tell an owner
which rule is broken -- raising there turned one bad row into a 500 that hid
every healthy rule in the namespace, and made a successful delete report
failure on the one call that would have cleaned it up. Ask what the caller will
do with the answer before choosing.
A cleanup step must not be abortable by its own instrumentation¶
psutil.Process.wait() documents NoSuchProcess and TimeoutExpired, and
ManagedExecutable._wait_for_exit() caught both. On Linux it waits with
pidfd_open() and falls back to its own waitpid() loop for exactly four
errnos, re-raising anything else as a bare OSError. A process which has been
SIGKILLed but not yet reaped can still be found by pid while no longer having a
thread group leader, and pidfd_open() answers EINVAL in that window rather
than ESRCH -- so roughly once in 1,600 kills on sfcbr, waiting for a dying
dnsmasq threw OSError out of ManagedExecutable.terminate(). The
configuration directory terminate() removes next was left behind, and the
network operation worker caught it as "Ignored error in net_op" and skipped the
rest of that network's node teardown (#4753).
Two lessons, and the second is the general one. Catch what a library's implementation can raise, not only what its documentation names. And a step added to make a teardown safer -- this wait exists because dnsmasq's replacement raced the dying process's sockets (#4472) -- must not be able to abort the teardown it was added to. Observing an exit is instrumentation; when instrumentation cannot answer, it says so and the cleanup still runs.
Releasing a lock is not part of the work¶
NodeLock.__exit__ was the whole unlock request and nothing else, so a
nodelock daemon which went away between the acquire and the release turned a
completed operation into a failed one. On sfcbr during an ordinary rolling
restart, _apply_delete_on_hypervisor ran its whole teardown, left the
with block, and then raised ConnectionResetError out of the unlock: the
instance was gone, the network teardown had run, and the delete operation was
marked failed anyway, so the client retried into an operation which found
everything already deleted (#4902). Three milliseconds later the enclosing
instance lock's release failed the same way.
A context manager's __exit__ has a second, worse failure mode: an exception
raised there replaces whatever the body raised. The real error becomes a
message about tidying up after it.
So a release failure is logged, not raised — ClusterLock.__exit__ has
swallowed its own LockNotHeld for this reason far longer. For a node lock
that is also safe, and the reason is worth stating because it is a property
of the daemon rather than of the lock: daemons/nodelock/main.py holds its
lock table as a plain dict local to main() and deletes a key before it
answers, so a daemon which cannot answer an unlock has either dropped every
lock it held or has already released this one. There is nothing leaked to
clean up.
The exception is a request which never arrived, because a nodelock has no
lease and so really would stay held until the daemon restarted — and a refused
connection is not proof the daemon is gone, since a healthy daemon with a full
listen backlog refuses connections too. __exit__ therefore retries the
unlock NODELOCK_UNLOCK_ATTEMPTS times before it gives up and logs.
UnknownNodeLockReplyException is deliberately outside
NODELOCK_TRANSPORT_ERRORS, the one list of what a restarting nodelock daemon
looks like to a client: a reply which parses but carries no known field is a
bug in this protocol, not a daemon blip, and it stays loud in __exit__ and in
health_check_nodelock() alike.
Fail closed on a field, not on a formatting accident¶
A reply that says "this went wrong" must say so in a field set only on the
success path. CountFederatedAttemptReply used to signal failure by carrying a
non-empty error, and the client read anything else as an answer -- so an
exception raised with no args, whose str() is empty, arrived as
attempts=0, error='' and was read as "nobody has tried this minute, allow".
That is the one direction a rate limiter must never fail, on the one endpoint
anybody may call.
Both federation replies now carry bool ok, set only where the work actually
succeeded, and the client tests that. error is diagnostics. When you add an
RPC whose reply has a permissive-looking default -- a zero count, an empty
list, a False that means "go ahead" -- carry an explicit success field rather
than inferring one, and write the test that returns the empty-error reply and
asserts the refusal. The invariant is not that today's code produces a message;
it is that no reply can be mistaken for a permissive one.
A state an object can leave is not a terminal state¶
A guard which asks "is this object gone?" must answer yes only for a state the
object cannot leave. The namespace delete handler used to count an instance in
error as already gone, alongside deleted. But error is where an instance
whose delete failed ends up, and that instance still holds its disks, its
interfaces and its placement. Its row is reachable only through its namespace,
so deleting the namespace over the top of it left a row with no owner and
nothing which would ever sweep it (issue 4847).
Worse, the refusal expired into permission. A failing delete walks
delete-wait -> delete-wait-error -> error, and only the middle state
blocked, so a caller who retried the namespace delete a moment later got
through over exactly the instance the first refusal had protected.
When a guard lists the states it lets through, check each one against the
state machine: can an object leave this state, and does it still hold anything
while it is in it? deleted is the only state which answers no to both.
Test the guard against every state on the failing path, not just the one the
guard was written for: each of those is a moment a caller can retry. See the
comment in AuthNamespaceEndpoint.delete() in shakenfist/external_api/auth.py.
or [] is a decision about what a failed read means¶
mariadb.get_objects_by_state() returns None when the read failed and []
when nothing matched, and says so in its docstring. Every or [] at a call
site erases that distinction, and the erasure is not neutral: it asserts that
the caller treats "we could not find out" and "there is nothing" the same way.
Sometimes that is true. Decide it deliberately, because for one caller in this
codebase it was catastrophically false.
get_active_blob_uuids() ended in or []. The cleaner uses its result as a
complement set -- it unlinks every blob file on disk whose uuid is not in the
list -- so a failed read arrived as "no blobs are active", which is an
instruction to delete the node's entire blob store. The trigger was not exotic:
an oversized GetObjectsByState reply (issue 3638) is a non-retryable
RESOURCE_EXHAUSTED, raised as-is by _grpc_call, mapped to None by the
client wrapper, and flattened to [] one line later. The same call in the
cluster daemon only ever iterates the list, so there the empty list was
merely a skipped pass. One accessor, two callers, opposite consequences.
So the rule is about the caller, not the accessor. Before collapsing an error
into a value, ask what each caller does with it. A caller that iterates can
usually tolerate a skipped pass -- catch explicitly, log, and move on. A caller
that complements, gates, or diffs against the list cannot, and must see the
failure. get_active_blob_uuids() now raises exceptions.DatabaseUnavailable,
which is the same shape as the fix for issue 3373 (an unreachable database must
not be indistinguishable from a missing object) and gets the REST path a clean
503 for free via handle_database_unavailable.
A sweep that reads a work list has a quieter version of the same problem: it
does not delete anything wrongly, it just does nothing, reports a healthy pass
over an empty queue, and lets the backlog grow -- which in issue 3638 grew the
very reply that could not be read. _sweep_work_list() in the cluster daemon's
scheduled tasks is the shared answer: a failed read returns None, is counted
into cluster_sweep_work_list_failure_streak, and is logged as a skipped pass.
The general form: silence is not success, and an empty result set is not the
absence of an answer.
One trap when handling this deliberately: a failed read from the mariadb
client arrives in two shapes. The client wrappers map grpc.RpcError to a
None/False/[] return, but _grpc_call() raises
exceptions.DatabaseUnavailable once its retry budget is spent, and that is
deliberately not an RpcError subclass (issue 3373), so it propagates through
the wrapper untouched. Handling only the return value covers the oversized-reply
case and misses the tier outage -- which is the more likely reason a read fails,
and the condition an alert on the streak most needs to see. Cover both, and
prove it with a test that sets side_effect = DatabaseUnavailable(...) rather
than a None return. Code which is meant to ride out an unreachable
database catches DatabaseUnavailable explicitly and says so;
Daemon.check_daemon_state() and ClusterLock.__enter__ are the
examples to copy.
Having covered both, do not then treat them alike inside a loop. The two shapes
differ in blast radius. A None return is per-reply, so the next object type is
still worth reading. A DatabaseUnavailable is tier-wide: every remaining read
will fail identically, and each will spend a full _grpc_call retry budget
before it does. Continuing therefore costs one budget per iteration, and both
the deleted-object sweep (28 object types) and orphan reconciliation run inside
a single scheduled job whose watchdog is only petted between jobs. The
tolerant continue that is right for the per-reply case would turn a database
outage into a SIGABRT of the elected cluster maintainer and a lock failover.
Continue on the per-reply shape; stop the loop on the tier-wide one. The
general form: when one read becomes N reads, recompute the caller's worst-case
wall time rather than inheriting it.
And when you stop a loop early, ask what it does to whatever is behind the stop.
Both of those loops iterate a fixed list of object types from the start of the
list every pass, so break on its own would mean a single persistently slow
type hides every type after it forever -- and the backlog that then accumulates
makes that type slower still, which is the #3638 ratchet rebuilt one level up.
They keep a resume offset and begin the next pass after whichever type stopped
the last one, so the pass stays bounded and every type is reached within a
bounded number of passes.
The same rule applies to the reply itself, one layer down. A gRPC reply whose
only payload is a repeated field has nowhere to put "the read failed", so
returning object_uuids=uuids or [] from a servicer hands the client the same
ambiguity the or [] at a call site does -- except the client cannot even see
that a decision was made. DatabaseServicer.GetObjectsByState did exactly this,
which left the blob-store deletion hazard fully reachable through the likelier
failure (MariaDB down while sf-database is up and answering) after the
client-side half had been fixed. Signal it on the status instead:
context.set_code(grpc.StatusCode.UNAVAILABLE) for a transient database error,
which _grpc_call retries and converts to DatabaseUnavailable, and INTERNAL
for a handler bug, which is non-retryable and so becomes a None return. When
you fix an empty-means-failure bug, check both ends of the wire.
And check every operand of the decision, not just the one that led you there. The cleaner's test is an or:
Hardening get_active_blob_uuids() fixed the first operand and left the store
exactly as deletable through the second, which reads the node's own blob
locations and flattened failure to [] at all three layers. Worse, the second
operand fails more often than the first, because a plain MariaDB
OperationalError -- a lock wait timeout, a deadlock, a dropped connection --
breaks it while sf-database itself stays healthy and answers everything else
normally. A guard on one input of a multi-input decision is not a guard on the
decision. Enumerate the inputs, and write the negative-control test for each.
That said, the fix is per-caller and not per-accessor, so it does not follow
that every accessor should raise. get_references_from() still collapses a
failed read to [], deliberately and with the reasoning in its docstring,
because every one of its callers iterates. The complement-set caller gets its
own raising accessor, get_node_blob_uuids(), built on the truthful internal
_get_references_from(). Two accessors over one read, named for what their
callers may assume, beats one accessor with a raise_on_failure flag that
every call site has to remember to set.
Cluster CI tests only run in the merge queue¶
The functional cluster lanes in Functional tests -- everything under
deploy/shakenfist_ci/cluster_ci_tests/ -- are skipped on pull_request and
run on merge_group. A green PR therefore says nothing about whether those
tests pass, or whether they can even reach their first assertion. Some lanes
do not run in the merge queue at all, only weekly: the Ubuntu base image, the
slim-tier topology with Kerbside, and a single machine (see
Merge lanes and weekly lanes). The
Ansible modules and Node lifecycle suites (ansiblemoduletests.sh and
nodelifecycletests.sh, shell scripts rather than cluster_ci_tests/
lanes) also run only weekly, so neither a pull request nor a retest in the
merge queue exercises them. To run any of those against a branch, dispatch
scheduled-tests.yml on it.
cluster_ci_tests/test_federation.py sat through four commits registering a
trusted issuer with an http:// jwks_uri while the API had refused
non-HTTPS jwks_uri since the object was added. Every test in the class died
in setUp on a 400, and nothing said so until the branch entered the queue.
Two habits follow. When you add or change a cluster CI test, run it, or at
minimum drive the validator it depends on directly -- these tests import
cleanly given pip install shakenfist-client testtools oslo.concurrency
prettytable in a scratch venv, so "the client is not installed" is not a
reason to skip verification. And when a test needs input that a validator
rejects, the validator is usually right: change the test, never carve a
loopback or test-only exemption into a security check. If that makes the test
unrunnable, make it skip loudly and file the issue -- see #3639 for the JWKS
certificate case.
That issue is a worked example of how the second habit tends to resolve. The
test needed the cluster to trust a certificate it had minted, and the tempting
fix was a test-only exemption in the deploy. What it became instead was
FEDERATION_JWKS_CA_BUNDLE -- extra trust anchors for JWKS fetches, which a
self hosted Authentik or Keycloak needs anyway -- with CI as its first user.
When a test cannot run because a security check is doing its job, the useful
question is usually "what would a real operator need here", not "how do we get
around this in CI".
Two traps it also left behind, both of which cost a debugging round:
ssl.create_default_context(cafile=...)replaces the system trust store rather than adding to it. Build the default context and then callload_verify_locationson it. The wrong spelling passes every test that only checks the new anchor is present.- Python 3.13 enables
ssl.VERIFY_X509_STRICTby default, so a leaf certificate with no Authority Key Identifier is refused with "Missing Authority Key Identifier". If you generate certificates in a test, give them AKI, SKI and basic constraints -- and be aware the symptom is indistinguishable from the CA not being trusted at all.
If a cluster test can run on one node, share it with the smoke suite¶
cluster_ci_tests/test_database_tier.py landed with two independent defects at
once (#3694, #3708) and blocked the merge queue for four days, because the
merge queue was the first place it ever ran. Both defects were reproducible on
a single node.
The two suites are disjoint directories -- smoke-ci.conf discovers only
smoke_ci_tests/, cluster-ci.conf only cluster_ci_tests/ -- so "add it to
smoke" cannot mean moving it without losing the multi-node coverage. Define the
test bodies once in a mixin under deploy/shakenfist_ci/ (not in either suite
directory, or stestr collects it twice from one run) and subclass the mixin
from both suites. database_tier.DatabaseTierTestsMixin is the worked example:
the two tests needing one sf-database run in both, and the load-balancer test
needing N>=2 stays in the cluster suite where its skip is honest.
Prefer this whenever a cluster test's preconditions are met by the single-node
localhost topology. Sometimes that topology is also the better test: the
API and sf-database share a machine there, so it is the most exposed to the
direct-MariaDB routing regression #3708 was about.
tearDown runs before addCleanup, and the base class already deletes¶
BaseNamespacedTestCase.tearDown() deletes every instance in the namespace and
blocks until they are gone, swallowing ResourceNotFoundException as it goes.
testtools runs tearDown() before the addCleanup stack:
So self.addCleanup(self.test_client.delete_instance, uuid) in a namespaced
test can only ever 404, and unlike tearDown's own deletes it is not guarded --
it fails the test after the assertions have passed. This is what kept
test_instance_get_fetches_the_attributes_row_once red even once its
measurement bug was fixed. Do not register instance cleanups in a namespaced
test; the base class already reaps them. Reserve addCleanup for state the
base class knows nothing about, such as the host devices in
test_stray_vxlan.py or a namespace the test made itself.
The last thing tearDown() does is delete the namespace, so anything
reached through a /auth/namespaces/<namespace>/... route -- a key, a
claim, a federation mapping rule -- has to be removed before
super().tearDown() is called, not from a cleanup. ClaimAPIMixin in
test_namespace_claims.py and TestLoki in test_loki.py are both
written that way, and both say why in a comment.
That was not true until 2026. _remove_namespace() asked whether a
namespace name was in the result of get_namespaces(), which is a list
of external_view() dicts, so the answer was always no and no namespaced
test deleted its namespace for six years. The rule above was correct
advice for the wrong reason, and it will be correct advice for the right
one only as long as the comparison stays fixed --
smoke_ci_tests/test_ci_harness.py is what holds it there. When a
helper's failure mode is to succeed quietly, the guard has to assert the
effect on the cluster rather than that the helper returned.
A signal handler only sets flags¶
Python runs a signal handler on the main thread, between any two of that
thread's bytecodes, so the handler shares every lock the main thread might be
holding at that instant -- and a non-reentrant one it needs means waiting for
itself forever. Logging takes one (the Loki handler enqueues under the logship
spool's lock, and the main thread logs all the time), and so does the
database client. sf-api's drain handler logged and deadlocked its worker until
gunicorn SIGKILLed it (#3495). Every daemon's exit_gracefully() logged and
wrote its state to the database, and sf-database, stopping under a slow
MariaDB, never got as far as its drain (#4601). Neither looked wrong.
A handler assigns, writes the abort file with daemon.write_abort_file(), or
calls a function marked @signal_safe (shakenfist/util/signals.py); the
main thread notices and does the real work. tools/check-signal-handlers.py
enforces this in pre-commit for every signal.signal() handler and, through
@signal_safe, every helper they call. When it cannot see that a call is
safe -- Event.set() on an event only another thread waits on -- end the
line with # signal-safe: <why>, and make the why something a reviewer can
check.
A column-to-column comparison has no bind processor¶
SQLAlchemy coerces a bound value through the column type on its way into the statement. It cannot do that for a comparison between two columns, because there is no Python value to coerce. Any join whose two sides store the same logical value in different physical forms therefore matches nothing, silently, and a query that returns no rows is indistinguishable from a feature with no work to do.
Both of object_states' join columns have that problem:
object_states.object_uuidis asa.Stringobject key holding the dashed uuid form (or, for a namespace, its name), while the static tables usesa.Uuid, which is undashed CHAR(32) on MariaDB. Join through_dashed_uuid_expr(), never a baresa.cast.object_states.object_typeissa.Enum(ObjectType), which SQLAlchemy stores by enum name ('NET_OP'), while the operation and object tables carry the enum value ('net_op'). Bind anObjectTypemember; do not compare the two columns.
The second is the subtler one, because binding a plain 'net_op' string
does work -- ObjectType subclasses str, so the member and its value
hash identically in the Enum type's _valid_lookup -- which makes every
other object_states query in mariadb.py correct by accident and makes
the broken join look like all of them.
The two coalescing primitives in mariadb.py had both defects in one
ON clause, each individually fatal, and shipped that way for three
months of green CI (issue #3878). Every test mocked either the primitive
or the engine, so nothing ever ran the statement against a table. The
operator-facing symptom was documented before the cause was: the
coalesced sibling ops event simply never fired. Both queries now share
_coalescible_states_join(), and shakenfist/tests/test_mariadb_coalescing.py
runs them against a real database.
The general rule: a query which can fail by matching nothing needs a test that proves it matches something. Asserting the statement's shape against a mocked engine cannot do that, and neither can a test that rewrites the query to suit sqlite -- register the missing SQL functions instead, so the statement under test is the statement that ships.
The same rule has a write-side form, and it cost longer to find. A test
which mocks the write helper and asserts it was called the right number
of times proves nothing about what the database kept. The v1
cluster_operation_targets table declared unique=True on
operation_uuid alone, so a multi-target operation's second and third
target rows raised IntegrityError, which the writer swallowed as
idempotency -- one row stored out of three. The test enqueued a hot plug
and asserted three calls with the right arguments, and passed, for months.
The symptom surfaced from the network side only: the first declared target
is the one that survives, so instance-side gates read correctly while
has_pending_cluster_operation() on the network returned False and the
network maintainer recreated a bridge an in-flight operation was about to
build. A constraint is part of the statement under test; a call count is
not a row count. What holds the invariant now is structural rather than a
test: every target travels as one targets= list into
create_and_enqueue_cluster_operation, written in the same transaction
as the operation, and the table's uniqueness is the composite
uq_cot_op_target on operation and target.
A uuid column's type is uuid_column_type(), never a bare sa.Uuid()¶
sa.Uuid() decides how to bind, read and render itself per dialect.
SQLAlchemy 2.0 treated MariaDB as having no native UUID type, so every
sa.Uuid() column became CHAR(32) holding undashed hex, and that is what
every existing Shaken Fist table contains. SQLAlchemy 2.1 made the MariaDB
dialect report native UUID support, and a bare sa.Uuid() stopped
converting: uuid.UUID binds reached the driver unconverted instead of as
the hex those CHAR(32) columns hold, reads came back as plain strings, and
newly created columns would have been MariaDB UUID. Nothing raised. The
coalescing keys on cluster_operations.network_uuid and node_uuid
simply matched no rows, and the SQLAlchemy 2.1 upgrade failed only in the
merge queue, as the coalescing cluster tests reporting the #3878 symptom
again.
shakenfist.schema.sqlalchemy.uuid_column_type() returns
sa.Uuid(native_uuid=False), which keeps the CHAR(32) behaviour on every
SQLAlchemy version. Use it for every uuid column, including the
SQLNativeUUID marker's (whose name predates this and is misleading: it
has never been a native UUID). shakenfist/tests/test_uuid_column_type.py
builds every table mariadb.py defines and fails on any sa.Uuid column
that could go native, and checks the bind, result and DDL behaviour
against the MariaDB dialect itself. The sqlite-backed coalescing tests
could not catch this: sqlite has no native UUID, so a bare sa.Uuid()
still behaves there exactly as it used to on MariaDB.
A frozen model is not a deep frozen model¶
mariadb.get_<type>() returns a Pydantic model with
model_config = ConfigDict(frozen=True), and that model is the one
entry the process-wide static object value cache holds for its whole
TTL. Every reader in the process gets the same object. frozen=True
stops attribute assignment on it; it does not stop mutation of the
contents of a list or dict field. So data.commands.pop(0) succeeds,
and it edits what every other reader on the node is about to see.
Two obligations follow, and they are two different obligations:
- Copy the container you hand out. A
_db_get()or_static_values_to_dict()copyinglist(data.commands)is the containment boundary. Without it, every object built from one cache hit shares one list, so a mutation anywhere is a mutation everywhere until the entry ages out. - Do not mutate an element of a container you were handed. The copy
above is shallow, which is deliberate -- a deep copy would defend
against a mutation no code performs and would cost an allocation per
element per read -- so the dicts inside a copied list are still the
cache's. Consume a command list by copying it and popping the copy,
the way
SideChannelExecutorJob.__init__does; build the dict you intend to edit locally, the wayadd_result()'s callers do.
SideChannelExecutorJob did the popping (issue #3516, PR #3970). It
aliased agentop.commands as its work queue and popped each command as
it dispatched, which drained the operation and, through it, the cache.
The visible damage was an inverted decision: operation_is_retryable()
opens by refusing an empty command list, so a single-command get-file
declared itself unretryable the instant its only command went out --
precisely the operation shape the retry path exists for. The second
consequence was quieter: NodeAgentopOp._preflight() iterates the same
list looking for put-blob commands whose blobs it must make local, and
against a drained cache entry it would find none and skip the
preparation with nothing logged.
Note what did not catch this. The class is covered by unit tests which
build the executor and drive a dispatch, but they hand-set
job.commands rather than going through the constructor, so the alias
was created in a line no test executed. A regression test for this class
of bug has to build the object the way production does and then assert
on what a second reader sees.
The static-values models with container fields today are
AgentOperationData.commands and InstanceData's disk_spec, video
and side_channels; all four are copied at their boundary. A new
container field on a cached model joins that list.
A per-item convenience wrapper does not belong in a loop¶
instance_usage_for_blob_uuid() answers "which instances use this
blob?". It does so by calling instance_blob_usage(), which walks
every healthy instance in the cluster: one FindInstances, then a
block_devices attribute read per instance and a get_references_from
per link of every disk's dependency chain. The wrapper is a single line
over a cluster-wide scan, and nothing about the call site says so.
Called once per request that is fine. Called once per item rendered it multiplies the whole scan by the length of the response, and the cost is invisible in the handler:
# Wrong: one cluster-wide instance walk per version.
for idx in ev['blobs']:
ev['blobs'][idx]['instances'] = instance_usage_for_blob_uuid(
ev['blobs'][idx]['uuid'])
# Right: one walk, indexed per version.
blob_usage = instance_blob_usage()
for idx in ev['blobs']:
ev['blobs'][idx]['instances'] = blob_usage.get(
str(ev['blobs'][idx]['uuid']), [])
Issue #3502 found this in the cluster cleanup loop and fixed it there,
and left a docstring on instance_blob_usage() saying which form to
use. Four API handlers kept the per-blob form anyway -- an artifact GET
walked once per version, the artifact and snapshot listings once per
artifact and once per version of every snapshot -- and it took issue
3876, filed against an unrelated symptom, to find them. The lesson is¶
not "read the docstring": it is that a function whose cost is proportional to the size of the cluster must be seen to be, so a per-item wrapper over one needs the bulk form to be the obvious thing to reach for and the loop body to be plainly an index lookup.
The same shape of trap exists wherever a property hides a query.
Blob.depends_on and Blob.transcoded each ran their own filtered
get_references_from while the caller had already read the same rows
unfiltered; that was the other half of #3876.
A no_log default is a cluster-wide redaction¶
An option declared no_log: true in a role's meta/argument_specs.yml is not
a per-task setting. Ansible registers whatever value the argument spec
validated as a secret for the rest of the run, and from ansible-core 2.22
every later occurrence of that exact string anywhere in the output -- task
names, assert messages, variables which have nothing to do with the option --
comes out as $REDACTED$.
The value validated is usually the role's own default, because an option an
operator did not set still gets validated. The node role defaulted
system_key, which the kerbside_credentials entry point declares no_log,
to "unknown", so a deploy which left it alone -- any deploy reading the key
from Vault, for one -- masked an ordinary English word out of every message of
the run. Upstream, the shakenfist.kerbside collection declared the whole
kerbside_sources list no_log, which registered each source's
type: shakenfist and turned a passing task header into
TASK [$REDACTED$.kerbside.kerbside : ...]; that failed the Kerbside wiring
test on every pull request in this repository until ansible-core was held
below 2.22 (issue #4773, kerbside#687). The role fix shipped in
shakenfist.kerbside 0.9.0, and the sanity job now pins ansible-core and
ansible-lint exactly, so a new release of either arrives as its own Renovate
pull request.
So the default of a no_log option must be a string which cannot occur
anywhere else: empty, which the masker ignores, or a change-me-<thing>
placeholder matching the example inventories. defaults/main.yml has to agree
with the spec, since it is the value a run actually validates.
test_collection_no_log_defaults.py checks both. Narrow no_log to the
fields which are secret rather than declaring a structure that contains one,
and remember that no_log on a task is a different thing -- it suppresses
that task's own output and registers nothing.
A backfill cannot finish while the old writers are still running¶
A migration which adds a column, fills it from somewhere else and then
advances the schema version has only filled the rows that existed when
it ran. On a rolling upgrade the schema moves first and the daemons
restart one at a time after it, so for the length of the deploy every
new row is written by code which does not know about the column and
takes its DEFAULT. Once the version reads the new number the
migration never runs again, and those rows keep the default forever.
event_objects.timestamp shipped exactly that way. Its v1 to v2
migration backfilled from events and its backfill was properly
restartable, which is what made it look complete; the rows it could not
have reached were the ones not yet written. One production deploy left
30,658 rows at timestamp = 0, and because the per-object read orders
on that column they sorted as the oldest events those objects had and
dropped out of the recent page (issue #4782).
So when a migration changes how rows are written, something which runs
after the restarts has to finish it. For event_objects that is the
first stage of the daily events prune sweep
(_direct_backfill_event_object_timestamps()), which repairs
default-valued rows in bounded batches and, once there are none, costs
one candidate query which finds nothing: a sweep that finds nothing is
the proof the deploy is over.
Re-running sf-ctl ensure-mariadb-schema is not an answer on its own,
because the version has already advanced and because an operator has to
remember to do it.
Two details worth copying. The repair must be first in a sweep with a
wall-clock budget, or a cluster whose other stages exhaust the budget
never reaches it. And it must not be a single table-wide UPDATE ...
JOIN: that next-key locks every row it examines while every node in the
cluster is trying to insert events, which is the defect
_direct_prune_events_by_type() already carries the measurements for.
A schema backfill runs in bounded batches¶
The previous rule's last paragraph says a repair must not be a single
table-wide UPDATE ... JOIN. Neither may the migration. sf-ctl
ensure-mariadb-schema runs from play 5 of the site playbook, before any
node restarts onto the new code, so every daemon in the cluster is
reading and writing the table a backfill is rewriting. Under REPEATABLE
READ one unbounded UPDATE next-key locks every row it examines, in
every table it touches, until it commits -- and a migration-time
backfill commits once, at the end.
The event_objects.timestamp backfill did that on a production cluster
and held 71.9 million row locks for 31 minutes. For all 31 of them every
node's event writes failed with MariaDB 1205 Lock wait timeout
exceeded (1,454 of them), clients saw 388 RecordEventBatch
DEADLINE_EXCEEDED errors, and object hard deletes -- which delete the
object's events -- waited on the same locks (issue #4797). Nothing was
wrong with the infrastructure and nothing in the migration failed; it
simply held its locks for as long as it took.
So a migration which rewrites more rows than fit comfortably in one
transaction calls _migration_backfill_batched(), which walks the table
in an indexed column's order and runs the backfill once per batch with
:cursor_from and :cursor_to bound to that batch's inclusive range.
Each batch is its own transaction, so a concurrent writer waits for one
batch at worst, and only if its row falls in the range currently held.
test_mariadb_migration_backfills.py reads the source of every
_ensure_*/_migrate_* function in mariadb.py and fails on an
UPDATE which is not bounded that way, because a new migration with
this defect would not fail any other test.
Two things to get right when you write one. The walk has to be a cursor, not a repeated "which rows still need work?" query: a candidate query scans past everything already done to find the next batch, which is quadratic in the size of the table and much slower than the statement it replaced. And the backfill statement keeps its own "not done yet" predicate alongside the range, which is what makes a batch idempotent and the whole migration restartable.
A sample taken before the work is not a decision¶
NodeInstOp._instance_delete() decided whether to tear a network's bridge
and vxlan device off the hypervisor from a list of "networks the node's other
instances use" that it built before it deleted the instance. Between the
sample and the teardown ran the whole of inst.delete() and one
update_dnsmasq() enqueue per network, and in that window an interface hot
plug or an instance create landing on the node attaches to one of those
networks. The sample says nobody else is using it, so a live domain loses its
network -- which does not announce itself as a race, it looks like the bridge
going missing for no reason (issue 4958).
The shape to copy was already in the same loop: it re-reads each Network
from the database and skips one in delete-wait rather than trusting
anything sampled earlier. So ask the question at the point of decision, or
hold across both the check and the act whatever lock the thing you are racing
takes. A sample is only as good as the window after it, and "it is only a few
statements later" is a statement about today's code.
Then, because the records are records and not observations, the host gets a
veto. util_network.foreign_bridge_members() asks ip which devices are
enslaved to br-vxlan-<vxid> and subtracts the ones Shaken Fist put there
itself; anything left is a guest tap, which is proof that a domain is
attached right now whatever the records say. The stray vxlan reaper
(daemons/network/maintain.py) has applied that veto before mutating for
longer, and the two share one definition of which devices are ours so they
cannot drift apart. A host which cannot be asked counts as in use, because
the two errors are not equal: a bridge left behind is reaped by a later
maintain pass, while a bridge taken from a live domain is an outage.