Consistency Audit Specifications

Every Shaken Fist project is expected to be packaged, documented, tested and automated the same way. This directory is the statement of what that means: one file per criterion, each defining what we check and why, and linking the template that implements it.

Who currently passes what is on compliance.md, regenerated every morning by the consistency audit and committed by it. It is the only generated file here; every criterion page beside it is hand-written, and each links its own section of the compliance page.

It sits under docs/ so that it publishes to shakenfist.com with everything else. What we hold a project to is documentation, and a criterion nobody outside the fleet can read is a criterion nobody outside the fleet can meet.

How audits work

Each file here is independently checkable, so an agent can be spawned per criterion to check every project against it in parallel.

docs/consistency-audits.md is the working reference for the system as a whole: what the daily run does, how issues are filed and closed, how the compliance tables are regenerated, how to add a criterion, and how to bring a repository into scope. Read it before adding a file here -- a new criterion is a check class and this specification, and a specification on its own does not measure anything.

File structure

Each audit file follows this structure:

# Audit: <name>

## What we check
<concise description of the audit criterion>

## Template
Template: `templates/<name>/`
See: `templates/<name>/README.md`

## Projects

Per-project compliance for this criterion is regenerated
every morning by the consistency audit: see
[the compliance page](/components/development/audits/compliance/#<name>).

The whole file is hand-written, including that link -- the anchor is the file's own basename, and the section it points at appears on compliance.md at the next run after the check is registered. Never put a consistency-audit marker block in a criterion page: the generated block carries a timestamp that moves daily, and a file carrying it can never hold a human review mark, which is why the tables were moved to one page in the first place.

In-scope projects

The following projects are subject to consistency audits:

• actions
• agent-python
• client-python
• client-python-k3s
• clingwrap
• cloudgood
• development
• divergulent
• hunkydory
• instar
• kerbside
• kerbside-client
• kerbside-patches
• library-utilities
• occystrap
• ryll
• shakenfist
• sfui
• uncalibrated-sextant
• visual-digest-rust

One project is in scope for part of the audit only:

• private-ci is audited for the plan-phase-references, plan-source-references, plan-index, plan-template and sfui-vendor checks, and nothing else. It is internal tooling and excluded from the conventions, but it vendors sfui and a vendored copy drifts silently: nothing in the consumer fails when the copy falls behind, or when someone edits it in place and the next sync discards the edit. It plans like the rest of the fleet too, so the four criteria that report a defect in a plan as written apply. The two that do not are plan-audit-phase, because the plans written there before it adopted the template track progress as checkpoint sections rather than as a phase table and enabling it would file an issue for a retrofit nobody has decided to do, and push-audit, which has no PUSH-AUDIT.md to read until that changes. It is not expected to grow a pyproject.toml, a renovate config, release workflows, or a develop branch.

Excluded projects

The following projects are excluded from these criteria, because they are internal only tooling, historical archive repositories, or not projects in the sense these criteria mean:

• ansible-modules
• client-js
• client-go
• client-python-ova
• deploy
• divergulent-reviews
• homebrew-tap
• images
• jenkins-private
• loadtest
• ostrich
• performance
• private-ci
• reproducables
• sonobouy
• symbolicmode
• terraform-provider-shakenfist
• uefi-latency-guest
• website

The actions repository is audited despite being tooling: the whole fleet depends on it for its composite actions and reusable workflows, so it is held to the same standards as anything else. development, this repository, is audited for the same reason turned around -- it is where these criteria and the tooling that enforces them are written, so an exemption here is one the authors of the standard write for themselves.

Some criteria do not apply to either. Neither has Python to package -- which takes pyproject-usage and python-version both out of scope -- and both keep main as their default branch: actions because every consumer pins to @main, development because it publishes no releases and so has no release branch for develop to integrate against.

All scoping and exemptions live in REPO_OVERRIDES in scripts/audit/repo.py, so they are reported as N/A with the reason attached rather than quietly disappearing from the table.

Audit index

File	Criterion
llm-tooling.md	AGENTS.md, ARCHITECTURE.md, Claude skills
llm-doc-structure.md	AGENTS.md and ARCHITECTURE.md are a summary and an index, detail lives in docs/
llm-context-lint.md	Agent context passes skillsaw at error severity, and every skill actually loads
llm-context-lint-ci.md	skillsaw runs in pre-commit and CI, not just in the daily audit
release-process.md	pyproject.toml, release.yml, RELEASE-SETUP.md
ci-review-automation.md	Automated review, developer automation workflows
renovate.md	Renovate for dependency bumps
renovate-lockstep-groups.md	Dependency families released in lockstep bump as one pull request
pin-indirect-dependencies.md	Pinning transitive dependencies
dependency-name-normalization.md	One spelling per pinned distribution
unused-declared-dependency.md	Declared dependencies are imported, or carry a recorded reason they are not
undeclared-direct-dependency.md	Nothing imported is left resting on a generated transitive pin
export-repo-config.md	Repo configuration export
default-branch-naming.md	Default branch conventions
github-security.md	Dependabot, secret scanning, CodeQL
delete-branch-on-merge.md	Branches are deleted automatically when a PR merges
merge-queue-config.md	Merge queues process entries serially, without speculative stacking or merge batching
merge-group-cancellation.md	Superseded merge group runs are cancelled, not left building clouds
security-sanitization.md	HTTP header and file path sanitization
workflow-standards.md	Permissions, naming, self-hosted runners, static runner tags, vm runner size, devpi cache fallback, devpi cache address, linting, PIPESTATUS, flake8wrap
expensive-lane-path-filter.md	Expensive PR lanes skip docs-only and review-marks-only changes
fuzz-nightly-reporting.md	Fuzz targets run nightly and file issues for crashes, rather than gating the merge queue
console-logging.md	Console script logging setup
python-version.md	Python version targeting and type hints
pyproject-usage.md	Python projects use pyproject.toml
version-file-gitignore.md	Generated version files are gitignored
rust-unwrap-lint.md	Rust projects lint against production unwrap()
readme-absolute-links.md	Top-level README.md links are absolute
docs-external-links.md	Links out of docs/ resolve inside docs/, or else are absolute
readme-structure.md	Top-level README.md is a pitch, detail lives in docs/
plan-phase-references.md	Docs describe current behaviour, not plan phase history
diagram-format.md	Diagrams of structure and flow are mermaid, not drawn in ASCII
mermaid-lint-ci.md	Repositories with mermaid diagrams render them in CI, so a broken one fails the build
plan-source-references.md	Plan references in source and configuration still resolve
plan-index.md	docs/plans/index.md layout, date ordering, plan coverage and the status vocabulary
plan-audit-phase.md	Master plans end with a phase that runs PUSH-AUDIT.md over the whole plan's work
push-audit.md	PUSH-AUDIT.md naming, versioned shared blocks, and an AGENTS.md reference to it
plan-template.md	PLAN-TEMPLATE.md shared blocks, including the sub-agent model roster and the push-audit phase
test-coverage.md	Unit and functional test coverage
secret-handling.md	Secret scanner in CI, credentials kept out of logs
review-coverage.md	Human review backlog stays under threshold in repos with review tracking
review-scope-completeness.md	Nothing leaves the human review queue by omission from the scope config
sfui-vendor.md	Vendored sfui copies are verbatim and current
scope-coverage.md	Every repository in the organisation is audited or documented as excluded
eol-distro.md	Nothing runs on, or is built from, a distribution release that has reached end of life

Beyond the audits

Everything here is a criterion because it can be stated plainly and, mostly, measured. That is not the whole of what we want from a project. We should be proud of our shared work: a regular holistic review of each project should ask what could be improved or tightened up with a refactor, and we should not be scared of a large refactor that delivers a large benefit -- while equally avoiding change for change's sake.

📝 Report an issue with this page