Consistency Audit Specifications

Every Shaken Fist project is expected to be packaged, documented, tested and automated the same way. This directory is the statement of what that means: one file per criterion, each defining what we check and why, and linking the template that implements it.

Who currently passes what is on compliance.md, regenerated weekly by the consistency audit and committed by it. It is the only generated file here; every criterion page beside it is hand-written, and each links its own section of the compliance page.

It sits under docs/ so that it publishes to shakenfist.com with everything else. What we hold a project to is documentation, and a criterion nobody outside the fleet can read is a criterion nobody outside the fleet can meet.

How audits work

Each file here is independently checkable, so an agent can be spawned per criterion to check every project against it in parallel.

docs/consistency-audits.md is the working reference for the system as a whole: what a run does, how issues are filed and closed, how the compliance tables are regenerated, how to add a criterion, and how to bring a repository into scope. Read it before adding a file here -- a new criterion is a check class and this specification, and a specification on its own does not measure anything.

Every criterion reads an audited repository the same way, so the specifications do not repeat it. A path that is not a regular file inside the repository once symlinks are followed -- a dangling link, a directory, a link pointing out of the clone -- is absent, and so is a file over 2 MiB, which is declined rather than judged on a prefix. A repository can therefore neither point the audit at a file it does not contain nor make it read without end.

File structure

Each audit file follows this structure:

# Audit: <name>

## What we check
<concise description of the audit criterion>

## Template
Template: `templates/<name>/`
See: `templates/<name>/README.md`

## Projects

Per-project compliance for this criterion is regenerated
on every run of the consistency audit: see
[the compliance page](/components/development/audits/compliance/#<name>).

The whole file is hand-written, including that link -- the anchor is the file's own basename, and the section it points at appears on compliance.md at the next run after the check is registered. Never put a consistency-audit marker block in a criterion page: the generated block carries a timestamp that moves every run, and a file carrying it can never hold a human review mark, which is why the tables were moved to one page in the first place.

In-scope projects

The following projects are subject to consistency audits:

• actions
• agent-python
• andris
• client-python
• client-python-k3s
• client-python-ova
• clingwrap
• cloudgood
• development
• divergulent
• hunkydory
• instar
• kerbside
• kerbside-client
• kerbside-patches
• library-utilities
• occystrap
• ryll
• shakenfist
• sfui
• uncalibrated-sextant
• visual-digest-rust

Two projects are in scope for part of the audit only:

• private-ci is audited for the plan-phase-references, plan-source-references, plan-index, plan-template, eol-producers and sfui-vendor checks, and nothing else. It is internal tooling and excluded from the conventions, but it vendors sfui and a vendored copy drifts silently: nothing in the consumer fails when the copy falls behind, or when someone edits it in place and the next sync discards the edit. It also owns the runner-label producer definitions eol-producers checks against the end-of-life table, so that applies too. It plans like the rest of the fleet too, so the four criteria that report a defect in a plan as written apply. The three that do not are plan-audit-phase, because the plans written there before it adopted the template track progress as checkpoint sections rather than as a phase table and enabling it would file an issue for a retrofit nobody has decided to do; push-audit, which has no PUSH-AUDIT.md to read until that changes; and plan-history-in-source, which comes in with push-audit, since that runbook carries the rule it measures. It is not expected to grow a pyproject.toml, a renovate config, release workflows, or a develop branch.

• images is audited for the eol-distro check, and nothing else. It is built fresh every night and the fleet's CI boots on its output, but it is excluded from the conventions because auditing it fully would file findings for conventions its own plan has not adopted, as a deliberate decision rather than a side effect of this one. eol-distro alone turns the compliance already confirmed by hand into a verdict the audit reports on a schedule, without manufacturing the rest.

Excluded projects

The following projects are excluded from these criteria, because they are internal only tooling, historical archive repositories, or not projects in the sense these criteria mean:

• ansible-modules
• client-js
• client-go
• deploy
• divergulent-reviews
• homebrew-tap
• images
• jenkins-private
• loadtest
• ostrich
• performance
• private-ci
• reproducables
• sonobouy
• symbolicmode
• terraform-provider-shakenfist
• uefi-latency-guest
• website

divergulent-reviews, homebrew-tap, performance, reproducables, sonobouy and uefi-latency-guest were confirmed on 2026-10-05 to have no .github/workflows/ directory, no container build file and no top-level templates/, so eol-distro would report not applicable for every one of them; that is why they stay excluded, which is narrower and truer than the reasons the list gives above.

The actions repository is audited despite being tooling: the whole fleet depends on it for its composite actions and reusable workflows, so it is held to the same standards as anything else. development, this repository, is audited for the same reason turned around -- it is where these criteria and the tooling that enforces them are written, so an exemption here is one the authors of the standard write for themselves.

Some criteria do not apply to either. Neither has Python to package -- which takes pyproject-usage and python-version both out of scope -- and both keep main as their default branch: actions because every consumer pins to @main, development because it publishes no releases and so has no release branch for develop to integrate against.

All scoping and exemptions live in REPO_OVERRIDES in scripts/audit/repo.py, so they are reported as N/A with the reason attached rather than quietly disappearing from the table.

Audit index

File	Criterion
llm-tooling.md	AGENTS.md, ARCHITECTURE.md, Claude skills
llm-doc-structure.md	AGENTS.md and ARCHITECTURE.md are a summary and an index, detail lives in docs/
llm-context-lint.md	Agent context passes skillsaw at error severity, and every skill actually loads
llm-context-lint-ci.md	skillsaw runs in pre-commit and CI, not just in the weekly audit
llm-doc-naming.md	Agent instructions live in AGENTS.md, not in a file named for one vendor
release-process.md	pyproject.toml, release.yml, RELEASE-SETUP.md
ci-review-automation.md	Automated review, developer automation workflows
renovate.md	Renovate for dependency bumps
renovate-lockstep-groups.md	Dependency families released in lockstep bump as one pull request
pin-indirect-dependencies.md	Pinning transitive dependencies
dependency-name-normalization.md	One spelling per pinned distribution
unused-declared-dependency.md	Declared dependencies are imported, or carry a recorded reason they are not
undeclared-direct-dependency.md	Nothing imported is left resting on a generated transitive pin
npm-pin-indirect-dependencies.md	package-lock.json is committed, lockfileVersion 2 or later, and honoured by CI
npm-unused-declared-dependency.md	Declared npm dependencies are imported, run, or named where they are used
npm-undeclared-direct-dependency.md	Nothing imported is left resting on a transitive npm resolution
export-repo-config.md	Repo configuration export
default-branch-naming.md	Default branch conventions
github-security.md	Dependabot, secret scanning, CodeQL
delete-branch-on-merge.md	Branches are deleted automatically when a PR merges
merge-queue-config.md	Merge queues process entries serially, without speculative stacking or merge batching
release-environment-protection.md	Every environment a release job names exists, has a required reviewer, and deploys only from tags
merge-group-cancellation.md	Superseded merge group runs are cancelled, not left building clouds
security-sanitization.md	HTTP header and file path sanitization
workflow-standards.md	Permissions, naming, self-hosted runners, static runner tags, vm runner size, devpi cache fallback, devpi cache address, linting, PIPESTATUS, flake8wrap
expensive-lane-path-filter.md	Expensive PR lanes skip docs-only and review-marks-only changes
fuzz-nightly-reporting.md	Fuzz targets run nightly and file issues for crashes, rather than gating the merge queue
scheduled-workflow-health.md	No workflow that fires unattended has failed without ever succeeding
reusable-workflow-secrets.md	Reusable workflows are passed named secrets, never secrets: inherit
console-logging.md	Console script logging setup
python-version.md	Python version targeting and type hints
pyproject-usage.md	Python projects use pyproject.toml
version-file-gitignore.md	Generated version files are gitignored
release-artifacts-exclude-plans.md	Release artifacts built from the repository root do not ship docs/plans
rust-unwrap-lint.md	Rust projects lint against production unwrap()
readme-absolute-links.md	Top-level README.md links are absolute
docs-external-links.md	Links out of docs/ resolve inside docs/, or else are absolute
readme-structure.md	Top-level README.md is a pitch, detail lives in docs/
plan-phase-references.md	Docs describe current behaviour, not plan phase history
diagram-format.md	Diagrams of structure and flow are mermaid, not drawn in ASCII
docs-line-references.md	Documentation names code by symbol, not by line number
mermaid-lint-ci.md	Repositories with mermaid diagrams render them in CI, so a broken one fails the build
plan-source-references.md	Plan references in source and configuration still resolve
plan-history-in-source.md	Source and configuration give the reason for the code, not the plan phase, decision or step that produced it
plan-index.md	docs/plans/index.md layout, date ordering, plan coverage and the status vocabulary
plan-audit-phase.md	Master plans end with a phase that runs PUSH-AUDIT.md over the whole plan's work
push-audit.md	PUSH-AUDIT.md naming, versioned shared blocks, and an AGENTS.md reference to it
plan-template.md	PLAN-TEMPLATE.md shared blocks, including the sub-agent model roster and the push-audit phase
test-coverage.md	Unit and functional test coverage
secret-handling.md	Secret scanner in CI, credentials kept out of logs
review-coverage.md	Human review backlog stays under threshold in repos with review tracking
review-scope-completeness.md	Nothing leaves the human review queue by omission from the scope config
sfui-vendor.md	Vendored sfui copies are verbatim and current
scope-coverage.md	Every repository in the organisation is audited or documented as excluded
eol-distro.md	Nothing runs on, or is built from, a distribution release that has reached end of life
eol-producers.md	No runner label is offered for a distribution release that has reached end of life

Beyond the audits

Everything here is a criterion because it can be stated plainly and, mostly, measured. That is not the whole of what we want from a project. We should be proud of our shared work: a regular holistic review of each project should ask what could be improved or tightened up with a refactor, and we should not be scared of a large refactor that delivers a large benefit -- while equally avoiding change for change's sake.

📝 Report an issue with this page