Authentication¶

Info

For a detailed discussion of how Shaken Fist authentication works, please see the discussion in the developer guide.

Terms used here are defined in the glossary.

Key expiry and reaping¶

Namespace keys may carry an optional expiry. Expiry is enforced when the key is used, so an expired key stops authenticating the instant it lapses -- there is no window during which it still works because a cleanup task has not run yet.

Tidying up expired keys is a separate, purely cosmetic concern. The cluster daemon sweeps every fifteen minutes and soft-deletes keys which expired more than NAMESPACE_KEY_REAP_GRACE seconds ago; the standard object reaper then hard-deletes them once they have been soft-deleted for CLEANER_DELAY. A key whose static row has no state row at all (a "zombie", see the orphan reconciliation section of the database guide) is skipped by this sweep — the hourly orphan reconciliation repairs it, after which the normal reap path removes it.

Setting	Default	Effect
`NAMESPACE_KEY_REAP_GRACE`	`3600`	Seconds after expiry before a key is soft-deleted. `0` disables reaping, retaining expired keys forever.

The grace period is there so that an operator investigating automation which suddenly stopped working can still see the key that lapsed. Lengthen it if you want a longer forensic window; set it to 0 if you would rather expired keys were never removed. Neither choice affects security, because enforcement does not depend on the sweep having run.

You cannot currently read an expiry back

A key's expiry is recorded on the key object, and no API returns it. GET /auth/namespaces/{namespace}/keys answers with a list of key names only, and the same is true of the scopes and the provenance of a key minted by the federated exchange.

This makes the forensic window above more important than it should be: when automation stops authenticating, the evidence that the key lapsed is the key's disappearance and its events, not a field you can query. Setting NAMESPACE_KEY_REAP_GRACE to 0 while investigating keeps that evidence indefinitely.

Tracked as issue #3672.

Cluster generated key secrets¶

Secrets Shaken Fist generates itself — the short-lived service keys used between nodes, keys minted by the federated exchange, and any key you ask the cluster to generate for you — carry a recognisable format:

sfk_<32 random characters><6 character checksum>

The prefix makes a leaked credential greppable in logs and repositories; the checksum lets a secret scanner reject lookalikes without calling the API. It costs nothing cryptographically, because a bearer credential is a random identifier rather than ciphertext, so the prefix is a label beside the random part rather than a revealed piece of it.

To have the cluster generate a key for you, create the key without supplying a secret. The generated secret is returned exactly once — only its bcrypt hash is stored, so it cannot be recovered afterwards.

This is a REST API feature which the command line does not expose yet: sf-client namespace add-key takes the secret as a mandatory argument, so there is no way to omit it. Until that changes, generation means curl:

curl -X POST https://sf.example.com/auth/namespaces/myproject/keys \
  -H "Authorization: Bearer ${SF_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"key_name": "deploy"}'
{"key_name": "deploy", "key": "sfk_..."}

The same applies to expiry, which is also body-only.

The prefix is reserved¶

sfk_ may not be used at the start of an operator-supplied key secret; attempting it is refused with a 400. This is not cosmetic. /auth rejects a presented secret which carries the prefix but fails its checksum before bcrypt comparing it against anything, and that shortcut is only sound if no legitimate operator secret can be shaped that way.

Upgrade note

If you have an existing key whose secret happens to begin with sfk_, it will stop authenticating after this upgrade, and it will fail as an ordinary 401 rather than with a distinctive error. A four-character prefix on a secret somebody chose makes this very unlikely, but it is not impossible. If an automation begins failing to authenticate immediately after upgrading and you cannot explain it, check whether its key starts with sfk_ and rotate it.

Key names are unaffected — only secrets.

Keys and the event log¶

Credentials are deliberately absent from Shaken Fist's events. This is worth knowing about because events are written to syslog and shipped to Loki, so anything in an event has left the cluster and is sitting in log aggregation, which usually has weaker access control than the credential does.

Events record the key name that was used, never the key secret, its stored hash, its nonce, or any minted or presented token. Separately, the API request tracing that records request and response bodies skips bodies entirely for routes under /auth, since those carry plaintext key secrets inbound and minted tokens outbound. The request URL is still logged, so you keep the namespace and the key name.

If you have log tooling that greps for tokens in Shaken Fist events, it will find nothing from these releases onward. That is the intent, not a regression.

Upgrading¶

Keys used to live in a JSON column on the namespace_attributes table and are now first-class objects in their own tables. Migration happens during sf-ctl ensure-mariadb-schema, preserving each key's hash, nonce and expiry exactly, so tokens minted before the upgrade keep validating and no operator action is required.

The migration is one-way in the usual sense: keys created after the upgrade are not written back to the old JSON column, so downgrading to a release that predates this change loses them. Keys that existed before the upgrade are unaffected either way. The exposure is one upgrade cycle, and it matches the precedent set by the node_daemon_states migration.

Federated identity¶

Workloads with an identity elsewhere -- a GitHub Actions job, a service account in an Authentik instance -- can trade that identity for a scoped, expiring namespace key rather than holding a long-lived Shaken Fist secret. The mechanics are in the developer guide; this covers what an operator has to decide and configure.

Trusting an issuer is an administrative act¶

A trusted issuer is cluster-wide, and creating one is a system namespace operation. It says the cluster will believe tokens this provider signs, so the decision belongs with whoever is responsible for the cluster rather than with an individual namespace owner.

Four fields, all mandatory:

Field	Notes
`name`	How rules refer to this issuer. Treat it as durable
`issuer_url`	Compared to a token's `iss` claim exactly: no normalisation, no trailing-slash tolerance
`jwks_uri`	Where the signing keys are published. Always taken from here, never from the token
`audience`	Tokens must be minted for this. Usually your cluster's API URL

The exact issuer_url comparison is deliberate. A loose comparison here is a way to accept tokens from somewhere else entirely, so if your tokens are refused with an untrusted-issuer message, check for a trailing slash before checking anything else.

Rules reference issuers by name

Deleting an issuer does not delete the mapping rules that name it; those rules simply stop working. Recreating an issuer under the same name silently rebinds every rule that named it, potentially to a different identity provider than the rules' authors intended. Renaming rather than recreating avoids this.

Delegating to namespace owners¶

A mapping rule is owned by the namespace it mints into, and creating one requires ownership of that namespace -- the same gate as adding a key, because a rule is the same privilege granted in advance and conditioned on claims. Once you have configured an issuer, namespace owners can write their own rules without coming back to you.

Two things to watch:

Rules targeting system. A rule that mints into the system namespace is a standing offer of administrative credentials to whoever satisfies its claims. This is permitted, because there are legitimate uses, but the cluster logs a warning and writes an audit event when such a rule is created. Those events are worth alerting on.
Scope breadth. A rule grants exactly the scopes it lists. artifact.* is a whole family; ["cluster-admin", "node.read"] is a genuinely least-privilege monitoring credential. Listing a rule's scopes is how you audit what a federated workload can do.

Listing a namespace's rules answers "who can get into this namespace", which is the inbound counterpart to listing its trusts.

A worked GitHub Actions example¶

GitHub can mint an OIDC token for a workflow job, describing the repository, the branch and the workflow that asked for it. This is the end to end shape of granting a repository's workflows scoped access to a namespace.

The sf-client command line does not wrap these routes yet, so the examples below call the REST API directly.

Configure the issuer once, as an administrator:

curl -X POST https://sf.example.com/auth/issuers \
  -H "Authorization: Bearer ${SF_ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "github",
        "issuer_url": "https://token.actions.githubusercontent.com",
        "jwks_uri": "https://token.actions.githubusercontent.com/.well-known/jwks",
        "audience": "https://sf.example.com"
      }'

Then, as the owner of the namespace the workflow should reach, write a rule saying which jobs qualify and what they get:

curl -X POST https://sf.example.com/auth/namespaces/ci/rules \
  -H "Authorization: Bearer ${SF_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{
        "name": "ryll",
        "issuer": "github",
        "bound_claims": {
          "repository": "shakenfist/ryll",
          "ref": ["refs/heads/develop", "refs/heads/main"]
        },
        "scopes": ["blob.read", "artifact.*"],
        "key_ttl": 3600,
        "key_name_prefix": "ryll-ci"
      }'

The workflow needs id-token: write permission to ask GitHub for a token, and nothing else:

permissions:
  contents: read
  id-token: write

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - name: Get a Shaken Fist key
        run: |
          IDENTITY=$(curl -sS \
            -H "Authorization: bearer ${ACTIONS_ID_TOKEN_REQUEST_TOKEN}" \
            "${ACTIONS_ID_TOKEN_REQUEST_URL}&audience=https://sf.example.com" \
            | jq -r .value)

          RESPONSE=$(curl -sS -X POST https://sf.example.com/auth/federated \
            -H "Content-Type: application/json" \
            -d "{\"token\": \"${IDENTITY}\",
                 \"namespace\": \"ci\",
                 \"rule\": \"ryll\"}")

          echo "::add-mask::$(echo "${RESPONSE}" | jq -r .key)"
          echo "SHAKENFIST_KEY=$(echo "${RESPONSE}" | jq -r .key)" >> "${GITHUB_ENV}"
          echo "SHAKENFIST_NAMESPACE=ci" >> "${GITHUB_ENV}"

The audience in the token request must match the issuer's configured audience exactly, and the add-mask line matters: the minted secret is a credential, and the response body is the only place it will ever appear.

There is no repository secret anywhere in this. The credential the job ends up holding is scoped to blob.read and artifact.*, expires an hour after it was minted, and its provenance records which rule minted it and which claims were satisfied -- so an audit of "what did that workflow have access to" is a query rather than an investigation.

Nothing here is specific to GitHub's hosted runners; a self hosted runner asks for its token the same way.

Abuse resistance¶

/auth/federated is unauthenticated by nature, so it carries its own protections.

Replay. An identity token is single-use per rule. The same token may still be exchanged through a different rule, so a workflow can reach two namespaces with one identity. Seen pairs live in the federation_replay table until the token they describe expires, and are reaped by the cluster daemon.

Rate limiting. Attempts are counted per source address per minute in the database, so the limit is cluster-wide rather than per API worker. Note that behind a reverse proxy which does not rewrite the source address, every request appears to come from the proxy and the limit becomes a single global one -- size it accordingly, or disable it and rate limit at the proxy instead.

Every attempt is counted, including ones refused for naming an issuer this cluster does not trust. Only the checks that read nothing but the request itself -- a missing field, an oversized body -- happen before the counter. So a misconfigured workflow pointed at the wrong issuer consumes its source's budget in the same way a wrong key does, which is worth knowing when a CI fleet sharing one NAT address starts seeing 429s.

Both checks fail closed: if the database cannot be reached the exchange answers 503 rather than assuming the request is fine.

Settings¶

Setting	Default	Notes
`FEDERATION_JWKS_CACHE_SECONDS`	300	How long an issuer's published keys are cached. Lower shortens the window in which a revoked key is still accepted; higher reduces load on the provider. An unknown key id always triggers an immediate refetch, so raising this does not delay recognising a rotated key
`FEDERATION_JWKS_FETCH_TIMEOUT_SECONDS`	5	How long to wait for an issuer's JWKS endpoint. The fetch happens while holding that issuer's refetch lock, so this is also the longest one unreachable provider can pin an API worker
`FEDERATION_MAX_TOKEN_BYTES`	16384	Largest exchange request accepted, refused before parsing. A real identity token is one to two kilobytes. A request with no `Content-Length` is refused with 411 rather than measured, so chunked encoding cannot opt out of the limit
`FEDERATION_RATE_LIMIT_PER_MINUTE`	60	Exchange attempts allowed per source address per minute. `0` disables rate limiting entirely
`FEDERATION_JWKS_CA_BUNDLE`	(empty)	Path to a PEM bundle of extra certificate authorities to trust when fetching an issuer's JWKS. Empty means the system trust store alone

An identity provider behind a private CA¶

A self hosted Authentik or Keycloak usually presents a certificate signed by the organisation's own CA rather than a public one, and the jwks_uri fetch will refuse it. Point FEDERATION_JWKS_CA_BUNDLE at a PEM file of the extra authorities to trust, and place that file on every node — any node may serve /auth/federated. The ansible collection's node role takes this as federation_jwks_ca_bundle; getting the file itself onto the nodes is yours to arrange, since it is usually the same bundle the rest of your fleet already has.

These anchors are added to the system ones rather than replacing them, so configuring a private provider does not stop a public one such as GitHub from verifying, and both can be trusted issuers at once.

Nothing else is relaxed by setting it. jwks_uri must still be https://, the hostname must still match, expiry is still checked, and a certificate chaining to neither set is still refused. There is deliberately no option to skip verification: a JWKS fetched over a connection nobody authenticated can be substituted by anyone on the path, which makes signature verification theatre.

If the path is wrong, or the file is not a PEM bundle, sf-api says so twice. Each worker complains at startup, so a bad path is a line in the log at deploy time rather than a surprise at somebody's first federated login weeks later. If it is still wrong when a token arrives, /auth/federated answers 503 rather than 401 — the token was never examined, and calling it rejected would send the caller to their identity provider to look for a fault which is in your config file. The log line names FEDERATION_JWKS_CA_BUNDLE and the path; the response deliberately does not.

If nobody uses it¶

Federation is inert until an issuer exists. A cluster which never creates one behaves exactly as it did before: the two tables stay empty, the reaper does nothing, and /auth/federated refuses everything with an untrusted-issuer message.

Trusts¶

Info

Trusts are a newer way of sharing between namespaces with granular control. If you instead are interested in making artifacts available to all users of a Shaken Fist cluster, then you should also consider artifact sharing, which is discussed in the artifacts section of the operators guide.

The system namespace is special in a Shaken Fist cluster in that it can see objects in all other namespaces. That is, if you are authenticated as the system namespace and list instances, you get not only the instances in the system namespace, but also all those in other namespaces. The same is true for other namespaced objects such as networks and artifacts.

In older versions of Shaken Fist this behavior was hard coded, but as of Shaken Fist v0.7 this is now implemented more flexibly. The system namespace must still be able to see every other namespace, but you can also create a "trust" relationship between two arbitrary namespaces to achieve the same result on a smaller scale. In fact, the system namespace is now simply a default trust that all other namespaces have a relationship with.

The Shaken Fist CI system uses these trusts for base images for CI runs. Each night we rebuild a series of base test images -- Debian 10, Debian 11, Ubuntu 20.04 and so on. Each Shaken Fist CI job is run in its own namespace, so we needed a place to store these base images, as well as a mechanism for other CI jobs to be able to see them.

What we implemented was:

a namespace to store the base images (we called it ci-images).
when our CI system creates a new CI runner and associated namespace, it creates a trust between that ephemeral namespace and the ci-images namespace.
jobs to create new images build them in their local namespace, and then "gift" them to the ci-images namespace via a label.
jobs which need to boot a test image can now see the images from the ci-images namespace by virtue of this trust relationship.

What a trust does and does not grant¶

A trust grants visibility over your resources — instances, networks, artifacts and the like. It is the system namespace's ability to see across namespaces, scaled down. A namespace you trust can list and read those objects. It cannot delete them, rename them, share them, or change their metadata — those all require the object's own namespace, or system.

Giving is a separate question from taking, and it is still allowed: a namespace you trust may create an object in your namespace, which is exactly the "gift" step in the ci-images example above. Creation is additive, you opted into it by extending the trust, and nothing you already had is lost by it. Deletion is none of those things.

Giving the same thing twice is not creation, though, and it is worth being precise about where the line falls. A trusted namespace may upload an artifact into yours when nothing of that source_url is there yet. It may not upload again over the artifact that first upload produced: adding a version ends in delete_old_versions, the caller supplies the blob, and the practical result is that an instance of yours booting that artifact afterwards gets somebody else's image. So the second upload is refused, and a CI job which pushes a nightly image into a shared namespace needs a key in that namespace rather than a trust.

Labels fall on the same side of that line, which they did not until v0.8. A label may be named <namespace>/<label>, so a caller can reach into your namespace through one, and nothing used to stop it — updating a label in your namespace did not even require a trust. A trusted namespace may now create a label in yours and may not update one that already exists, matching artifacts exactly. This does affect the ci-images pattern above: the first gift of a given label works, and a job which republishes under the same label name every night needs a key in the receiving namespace.

Namespace administration is the exception, and it is a large one. Adding a key to your namespace, and writing a mapping rule in it, are both gated on this same trust relationship rather than on ownership. A namespace you trust can therefore mint credentials in yours — directly with add-key, or by writing a mapping rule, which is a standing offer to mint a key for anyone holding a matching identity token. It can also delete the rules you wrote.

So a trust is not only a window onto your resources; it is administrative access to your namespace's credentials. Extend one to a namespace you would be willing to hand a key to, and read Federated identity before extending one to a namespace that has mapping rules configured.

Note

Artifacts behaved differently until recently: a trusted namespace could delete another namespace's artifacts. Instances and networks never permitted this, and artifacts now match them. If you have tooling which relied on deleting artifacts across a trust, it needs a key in the owning namespace, or the system namespace, instead.

📝 Report an issue with this page