CI Review Automation¶

Shaken Fist projects use Claude Code-powered automation for PR reviews, test fixing, and comment addressing. This page describes the workflow templates and how to add them to a new project.

How It Works¶

The automation consists of several GitHub Actions workflows that respond to PR events and bot commands:

flowchart TD
    pr[PR opened or updated] --> ci[CI tests run<br/>functional-tests.yml]
    ci -->|tests pass| review[Automated reviewer<br/>Claude Code]
    review --> comment[Posts structured review comment]

The review is where the automation stops. Its findings are worked through interactively; the workflow which acted on them automatically is retired, see below.

Bot Commands¶

Repository collaborators with write access can trigger these commands by commenting on a PR:

Command	Workflow	Description
`@shakenfist-bot please retest`	`pr-retest.yml`	Re-run functional tests
`@shakenfist-bot please re-review`	`pr-re-review.yml`	Fresh automated review
`@shakenfist-bot please attempt to fix`	`pr-fix-tests.yml`	Fix failing tests (separate template)

Repeating a command while the first one is still running does not run it twice. Both pr-re-review.yml and pr-retest.yml group their work per pull request and cancel the earlier run, so the last request wins and the run that cancelled its predecessor posts its own confirmation.

Which commit a re-review sees¶

A re-review checks out GitHub's merge of the pull request into its base, so the reviewer judges the change where it will land rather than in isolation. GitHub does not always publish a merge commit for a pull request, and -- because it recomputes one asynchronously after a push -- the one it publishes is not always for the current head. Reviewing a stale merge commit produces a careful review of superseded code, and nothing about the run looks wrong.

So pr-re-review.yml resolves the merge ref rather than naming it: it confirms the published merge commit is a merge of this pull request's head, waits briefly for a recomputation to catch up, and otherwise falls back to reviewing the head on its own. When it falls back it says so in a comment on the pull request, naming which of the two cases it hit. A review of the head is worth having; a reader who believes it saw the merge result is not.

Security Model¶

These workflows use issue_comment triggers, which run with elevated permissions. Security is enforced through multiple layers:

Authorization -- only repository collaborators with write access can trigger commands (enforced by shakenfist/actions/pr-bot-trigger)
Trusted tools -- scripts are checked out from the base branch, not the PR, preventing execution of malicious PR code
No credential persistence -- persist-credentials: false prevents tokens from being stored in the checkout
Git hooks disabled -- the workflows that put a pull request's code on the runner (pr-re-review.yml, and test-drift-fix.yml behind pr-fix-tests.yml) set core.hooksPath=/dev/null on the checkout before running anything over it. pr-retest.yml never checks out pull request code at all. The setting is repository local rather than --global, so it cannot affect any other repository's jobs on the same runner -- these jobs run on the shared, long-lived claude-code pool, not on a throwaway VM. This is a rollout in progress: the setting is in the templates and in this repository, but the repositories that copied pr-re-review.yml before it existed do not have it, and check_ci_review_automation does not yet look for it, so the daily audit will not tell you which ones
No pre-commit -- pre-commit hooks execute repository code and are skipped in privileged workflows. test-drift-fix.yml is the deliberate exception: fixing test drift means running the pull request's own tests, and its prompt asks Claude to run pre-commit run --all-files too, so the pull request's hooks do execute in a privileged job. core.hooksPath does not cover that -- it stops git-invoked hooks, not an explicit pre-commit run. That workflow rests instead on maintainer-only triggering and runner isolation, as its own file header states
Just-in-time auth -- gh auth setup-git is used only when pushing, not during the entire workflow

See the GitHub Security Lab article for background on issue_comment trigger security.

Workflow Templates¶

Templates are in templates/ci-review-automation/:

Template	Customisation	Description
`pr-re-review.yml`	None	Manual re-review trigger
`pr-retest.yml`	None	Manual test re-run

Both files are project-agnostic and can be copied directly.

For projects with large test suites that would benefit from automatic test fixing, see the separate templates/test-drift-fix/ templates which provide pr-fix-tests.yml and test-drift-fix.yml.

Adding CI Review Automation to a Project¶

Step 1: Copy the Workflow Files¶

# From the target project root:
cp /path/to/development/templates/ci-review-automation/pr-re-review.yml \
    .github/workflows/
cp /path/to/development/templates/ci-review-automation/pr-retest.yml \
    .github/workflows/

For projects with large test suites, also copy from templates/test-drift-fix/:

cp /path/to/development/templates/test-drift-fix/pr-fix-tests.yml \
    .github/workflows/
cp /path/to/development/templates/test-drift-fix/test-drift-fix.yml \
    .github/workflows/
# Then customise test-drift-fix.yml for your project

Step 2: Add Automated Reviewer to CI¶

Modify your main CI workflow (e.g. functional-tests.yml) to add:

A top-level permissions block with pull-requests: write
A job calling the shared shakenfist/actions/.github/workflows/pr-auto-review.yml@main, with the project's test jobs in its needs: list

See the template README for the exact YAML snippets.

Step 3: Ensure Runner Labels¶

Your self-hosted runners need these labels:

claude-code -- runners with Claude Code CLI installed
static -- small runners for non-mutating jobs (bot trigger parsing, permission checks)

Not Reviewing The Bot's Own Commits¶

A push authored by bot@shakenfist.com skips the automated reviewer. Projects calling the shared pr-auto-review.yml get this for free and should delete any local check-bot-commit job; see Not Reviewing The Bot's Own Commits for what the guard is for.

The retired comment addresser¶

There is no pr-address-comments.yml template here, and there will not be one: the comment addresser is retired, and a repository still carrying any part of it fails the audit. What it was, why it went, and exactly which files to delete are in docs/audits/ci-review-automation.md.

One thing about the reaping belongs here, because it is about the templates rather than about the audit. render-review.py in the shared action still ends every review it posts with a line telling the reader to use the addresser's trigger phrase. Once the chain is reaped that invites a command nothing answers -- no workflow, no reply, no failure -- which is the outcome the retired workflow's own failure reporting existed to avoid. Dropping those lines is a change to shakenfist/actions and cannot land here.

Shared Actions¶

The trigger and review logic lives in the shakenfist/actions repository:

pr-bot-trigger -- parses @shakenfist-bot commands, checks permissions, adds reactions, posts status messages
review-pr-with-claude -- runs automated code reviews with structured JSON output and embedded review data

Projects Using This Automation¶

Which projects have which of these is measured every morning rather than listed here, because a hand-maintained table of fleet state goes stale silently: see the ci-review-automation section of the compliance page.

📝 Report an issue with this page