CI Review Automation¶

Shaken Fist projects use Claude Code-powered automation for PR reviews, test fixing, and comment addressing. This page describes the workflow templates and how to add them to a new project.

How It Works¶

The automation consists of several GitHub Actions workflows that respond to PR events and bot commands:

flowchart TD
    pr[PR opened or updated] --> ci[CI tests run<br/>functional-tests.yml]
    ci -->|tests pass| review[Automated reviewer<br/>Claude Code]
    review --> comment[Posts structured review comment]

The review is where the automation stops. Its findings are worked through interactively; the workflow which acted on them automatically is retired, see below.

Bot Commands¶

Repository collaborators with write access can trigger these commands by commenting on a PR:

Command	Workflow	Description
`@shakenfist-bot please retest`	`pr-retest.yml`	Re-run the test workflow
`@shakenfist-bot please re-review`	`pr-re-review.yml`	Fresh automated review
`@shakenfist-bot please attempt to fix`	`pr-fix-tests.yml`	Fix failing tests (separate template)

Repeating a command while the first one is still running does not run it twice. Both pr-re-review.yml and pr-retest.yml group their work per pull request and cancel the earlier run, so the last request wins and the run that cancelled its predecessor posts its own confirmation.

Which commit a re-review sees¶

A re-review checks out GitHub's merge of the pull request into its base, so the reviewer judges the change where it will land rather than in isolation. GitHub does not always publish a merge commit for a pull request, and -- because it recomputes one asynchronously after a push -- the one it publishes is not always for the current head. Reviewing a stale merge commit produces a careful review of superseded code, and nothing about the run looks wrong.

So pr-re-review.yml resolves the merge ref rather than naming it: it confirms the published merge commit is a merge of this pull request's head, waits briefly for a recomputation to catch up, and otherwise falls back to reviewing the head on its own. When it falls back it says so in a comment on the pull request, naming which of the two cases it hit. A review of the head is worth having; a reader who believes it saw the merge result is not.

On both paths the workflow then confirms that the tree it checked out is the commit it validated -- the merge's second parent on the merge path, HEAD itself on the fallback path -- because a push between resolving the ref and checking it out moves what the ref reaches. If the head moved in that window the run fails rather than reviewing code nobody validated, and the requester comments again to review the new head.

Security Model¶

These workflows use issue_comment triggers, which run with elevated permissions. Security is enforced through multiple layers:

Authorization -- only repository collaborators with write access can trigger commands, and only on pull requests whose head is in this repository rather than a fork (enforced by shakenfist/actions/pr-bot-trigger, which folds both into its authorized output)

The work job in pr-re-review.yml and pr-retest.yml then requires the trigger job's same_repo export to be 'true' as well as authorized. That is deliberately redundant: callers take the action at @main, so a regression there would otherwise widen what runs next to a write-scoped token without any change in the calling repository. The trigger job also fails if the action stops reporting the outputs it reads, rather than letting the work job skip silently. The ci-review-automation criterion measures both halves of the gate on every adopter. 2. Trusted tools -- scripts are checked out from the base branch, not the PR, preventing execution of malicious PR code 3. No credential persistence -- persist-credentials: false prevents tokens from being stored in the checkout 4. Git hooks disabled -- the workflows that put a pull request's code on the runner (pr-re-review.yml, and test-drift-fix.yml behind pr-fix-tests.yml) set core.hooksPath=/dev/null on the checkout before running anything over it. pr-retest.yml never checks out pull request code at all. The setting is repository local rather than --global, so it cannot affect any other repository's jobs on the same runner -- these jobs run on the shared, long-lived claude-code pool, not on a throwaway VM. This is a rollout in progress: the setting is in the templates and in this repository, but the repositories that copied pr-re-review.yml before it existed do not have it, and check_ci_review_automation does not yet look for it, so the weekly audit will not tell you which ones 5. No pre-commit -- pre-commit hooks execute repository code and are skipped in privileged workflows. test-drift-fix.yml is the deliberate exception: fixing test drift means running the pull request's own tests, and its prompt asks Claude to run pre-commit run --all-files too, so the pull request's hooks do execute in a privileged job. core.hooksPath does not cover that -- it stops git-invoked hooks, not an explicit pre-commit run. That workflow rests instead on maintainer-only triggering and runner isolation, as its own file header states 6. Just-in-time auth -- gh auth setup-git is used only when pushing, not during the entire workflow

See the GitHub Security Lab article for background on issue_comment trigger security.

Workflow Templates¶

Templates are in templates/ci-review-automation/:

Template	Customisation	Description
`pr-re-review.yml`	None	Manual re-review trigger
`pr-retest.yml`	`RETEST_WORKFLOW` repository variable, if the test workflow is not `functional-tests.yml`	Manual test re-run

Both files are project-agnostic and can be copied directly. pr-retest.yml dispatches functional-tests.yml unless the RETEST_WORKFLOW repository variable names another workflow; see the template README.

For projects with large test suites that would benefit from automatic test fixing, see the separate templates/test-drift-fix/ templates which provide pr-fix-tests.yml and test-drift-fix.yml.

Adding CI Review Automation to a Project¶

Step 1: Copy the Workflow Files¶

# From the target project root:
cp /path/to/development/templates/ci-review-automation/pr-re-review.yml \
    .github/workflows/
cp /path/to/development/templates/ci-review-automation/pr-retest.yml \
    .github/workflows/

For projects with large test suites, also copy from templates/test-drift-fix/:

cp /path/to/development/templates/test-drift-fix/pr-fix-tests.yml \
    .github/workflows/
cp /path/to/development/templates/test-drift-fix/test-drift-fix.yml \
    .github/workflows/
# Then customise test-drift-fix.yml for your project

Step 2: Add Automated Reviewer to CI¶

Modify your main CI workflow (e.g. functional-tests.yml) to add:

A top-level permissions block with pull-requests: write
A job calling the shared shakenfist/actions/.github/workflows/pr-auto-review.yml@main, with the project's test jobs in its needs: list

See the template README for the exact YAML snippets.

Step 3: Ensure Runner Labels¶

Your self-hosted runners need these labels:

claude-code -- runners with Claude Code CLI installed
static -- small runners for non-mutating jobs (bot trigger parsing, permission checks)

Not Reviewing The Bot's Own Commits¶

A push authored by bot@shakenfist.com skips the automated reviewer. Projects calling the shared pr-auto-review.yml get this for free and should delete any local check-bot-commit job; see Not Reviewing The Bot's Own Commits for what the guard is for.

The retired comment addresser¶

There is no pr-address-comments.yml template here, and there will not be one: the comment addresser is retired, and a repository still carrying any part of it fails the audit. What it was, why it went, and exactly which files to delete are in docs/audits/ci-review-automation.md.

One thing about the reaping belongs here, because it is about the templates rather than about the audit. render-review.py in the shared action still ends every review it posts with a line telling the reader to use the addresser's trigger phrase. Once the chain is reaped that invites a command nothing answers -- no workflow, no reply, no failure -- which is the outcome the retired workflow's own failure reporting existed to avoid. Dropping those lines is a change to shakenfist/actions and cannot land here.

Shared Actions¶

The trigger and review logic lives in the shakenfist/actions repository:

pr-bot-trigger -- parses @shakenfist-bot commands, checks permissions, adds reactions, posts status messages
review-pr-with-claude -- runs automated code reviews with structured JSON output and embedded review data

Projects Using This Automation¶

Which projects have which of these is measured weekly rather than listed here, because a hand-maintained table of fleet state goes stale silently: see the ci-review-automation section of the compliance page.

📝 Report an issue with this page