Development¶

How to build, test, and contribute to instar. See AGENTS.md for conventions and the Claude Code skills, and testing.md for the integration test suite in detail.

Building instar¶

# Build the main instar project
make instar

# The binaries will be in src/target/release/
sudo src/target/release/instar info <IMAGE>
sudo src/target/release/instar copy <INPUT> <OUTPUT>

Build and dev containers¶

The build runs in Docker, and there are two devcontainer images:

instar-release — a minimal debian:bullseye image (src/.devcontainer/build/Dockerfile) carrying only the toolchain that produces the release artifacts: the C linker, protobuf-compiler, the pinned Rust nightly with rust-src + llvm-tools, cargo-binutils, cargo-deb, cargo-generate-rpm. Used by make instar, make deb, make rpm. It is built on bullseye deliberately: glibc is forward-compatible, so building the host binary against glibc 2.31 lets one artifact run on every distro down to Rocky/RHEL 9 and Ubuntu 22.04 (see installation.md).
instar-build — the full Debian dev/test image (src/.devcontainer/Dockerfile, base pinned by digest) with qemu-utils, the libyal parsers, cargo-fuzz, cargo-audit, and gh. Used by everything else: make test, make test-rust, the make test-container* targets, make audit, the fuzz targets, and the VS Code devcontainer.

make clean-devcontainers removes both. To prove the release binary's glibc floor empirically, tools/verify-glibc-floor.sh <deb> <rpm> installs the packages on every target distribution and runs info/create/map under KVM.

Do not bump the release image's base¶

src/.devcontainer/build/Dockerfile pins debian:bullseye, and that pin is the product rather than an implementation detail: the base's glibc is the floor of every binary we ship, so a newer Debian is a regression dressed as an update. renovate.json excludes the file for that reason; the dev/test image next door stays managed normally.

Two checks back the pin up, because a comment is not a control:

tools/ci/check-glibc-floor.sh runs on the line immediately after make instar, in both build-and-test and the release workflow, and fails if the binary references a symbol above GLIBC_2.31 — the floor published in installation.md, not the matrix CI's oldest distro (Rocky 9, 2.34), which is looser than the promise and would let a smaller base movement through unnoticed. It needs nothing but the built binary, so it gates every pull request. The placement is deliberate: the unit test run that follows builds bin targets into the same src/target/release/ from the dev image, so a later check could read a binary that image relinked.
tools/verify-glibc-floor.sh is the empirical version above, and remains the real acceptance gate — but it needs containers, packages and /dev/kvm, so it runs in the merge queue matrix or by hand.

The gap between those two is not hypothetical. Renovate raised the base to debian:trixie in #488; the floor went from GLIBC_2.30 to GLIBC_2.39; pull request CI passed completely, because nothing on the pull request path looks at the floor; and the failure surfaced as three red distros in the merge queue after the change was already on develop. The cheap check exists to make that a pull request failure.

bullseye reached end of life: apt is pinned to a snapshot¶

Debian 11 LTS ended on 2026-08-31 (wiki.debian.org/LTS), and it broke the image within a week, exactly as predicted: not as a glibc change but as an apt-get failure on a runner with no cached instar-release, while every warm runner stayed green. First the deb.debian.org CDN began returning 404 for pool objects the bullseye-security index still advertised (gnupg2, glibc, xz-utils), then the final bullseye-security Release expired at 2026-09-07 21:13 UTC, after which apt rejects the suite outright. See #550.

The decision taken was the snapshot pin. src/.devcontainer/build/Dockerfile still says FROM debian:bullseye, but rewrites /etc/apt/sources.list to a frozen snapshot.debian.org timestamp (ARG DEBIAN_SNAPSHOT, currently 20260901T000000Z — just after the last bullseye-security publication, so the image carries the final security state of the suite) and sets Acquire::Check-Valid-Until "false", because a snapshot serves the historical Release file including its long-past Valid-Until. gpg signature verification stays on; the sources stay http:// because the base ships no ca-certificates and that is the layer installing it. archive.debian.org was not usable: it carries debian/dists/bullseye but still has no debian-security/dists/bullseye-security.

The cost is a build toolchain that no longer receives updates. That is accepted deliberately — the shipped binary's security posture rests on the Rust toolchain (pinned, and bumped weekly by rust-nightly-bump.yml) and on our own code, not on the build image's C library, none of which is linked into the artifact beyond glibc itself.

To move the pin, change DEBIAN_SNAPSHOT only. Do not change FROM debian:bullseye, and do not "fix" a future apt failure by pointing back at deb.debian.org.

The options that were not taken¶

No option was free, because "move to the next Debian" is not available: bookworm ships glibc 2.36, above both the 2.31 we publish and the 2.34 the matrix needs, so taking it silently drops Debian 11, Ubuntu 22.04 and Rocky 9.

Move to bookworm and narrow the promise to glibc 2.36. This drops Debian 11, Ubuntu 22.04 LTS and Rocky/RHEL 9 — the last of which is in the matrix CI and, per PLAN-distro-matrix-ci.md decision D1, was the reason the floor was set this low. It needs installation.md, README.md, MAX_GLIBC in check-glibc-floor.sh, and the matrix distro list all updated together.
Change build strategy — a zig cc or cargo-zigbuild style cross-link against an explicitly chosen older glibc, decoupling the floor from the base image entirely. The most work, and the only option that stops this recurring every few years.

Pre-commit hooks¶

This project uses pre-commit hooks for Rust code quality:

# Install pre-commit (if not already installed)
pip install pre-commit

# Install the hooks
pre-commit install

# Run manually on all files
pre-commit run --all-files

The hooks run rustfmt (formatting) and clippy (linting) on all Rust code via Docker, ensuring consistent tooling regardless of local Rust installation.

Beyond the Rust hooks, pre-commit run --all-files also runs:

skillsaw, which lints the agent context — AGENTS.md, CLAUDE.md and the skills and settings under .claude/ — for malformed frontmatter, instructions smuggled into a file an agent is handed, embedded credentials, and dangerous hook or settings configuration. The hook is pinned by rev in .pre-commit-config.yaml and fetches its own environment on first run, so the first invocation after a fresh checkout is slower than the rest. The same hook runs in CI as the Agent context check.
actionlint over .github/workflows/, and shellcheck over scripts/ and tools/.
binary-sizes, which checks the built binaries against the guest memory layout, and devcontainer-pins, which fails if the two devcontainer Dockerfiles disagree about the Rust nightly or about the version of any cargo tool they both install, if a cargo install line has lost its --locked, or if a version pin has stopped being visible to Renovate. See "Cargo tool pinning" below.

To auto-fix formatting issues:

./scripts/check-rust.sh fix

Diagrams¶

Diagrams of structure or flow -- components and the arrows between them, an ordered exchange of messages, a state machine -- are written as fenced mermaid blocks, which GitHub renders natively. Character art that is not a diagram stays in a plain fence: file trees, memory maps, register and bit-field layouts, wire-format byte layouts and captured terminal output all carry their meaning in the column alignment, and mermaid would destroy it. The full policy is the diagram-discipline shared block in PUSH-AUDIT.md.

Mermaid fails at render time rather than at commit time, so a syntax error commits cleanly and then shows an error box on GitHub. Render before you push:

tools/mermaid-lint.sh            # every tracked markdown file
tools/mermaid-lint.sh docs/x.md  # just this one

Check the exit status directly. Piping the script into tail or grep reports the filter's status, not the script's, and turns every failure green. tools/audit/wave1.sh runs it as part of the pre-push audit, and CI runs the same script from mermaid-lint.yml on any pull request that touches markdown, on any push to develop or main that does, and on demand via workflow_dispatch. The lane gates nothing automatically, so without the push trigger a commit that lands without a pull request would go unlinted until it failed somebody else's markdown change. (The template ships both branch names because the fleet's default branch varies; here only develop ever matches.)

mmdc reads exactly one fence: three backticks, then mermaid, then nothing. GitHub renders a wider family, and every shape in between is rejected rather than skipped -- a tilde fence, a space before the language, four or more backticks, and anything after the language such as ```mermaid title=x. The script names the file, the line and what to change. Trailing whitespace after the language is fine; mmdc reads that. Quoting one of the rejected forms in a document is also fine -- a fence nested inside a longer fence is an example, not a diagram, and this paragraph's own page is the case that rule exists for.

Put a diagram at the top level. A mermaid fence inside a blockquote is the one shape that is neither linted nor rejected: GitHub renders it and mmdc finds nothing in the file, so it passes silently. That is a deliberate blind spot rather than an oversight -- refusing one means ruling on a fence nested inside a blockquoted fence -- but it does mean a blockquoted diagram is unchecked.

If this repository ever grows a REVIEWS.md, the script skips it, matching the workflow's path filter: a review session rewrites the file and changes no diagram. Name it as an argument to lint it anyway.

The container runs with --network none, so a diagram that reaches for a remote font or icon pack fails here rather than rendering differently depending on the runner.

Makefile¶

A Makefile is provided for common development tasks:

# Show all available targets
make help

# List available prototypes
make list-prototypes

Main Instar Project:

# Build instar
make instar

# Clean instar build
make clean-instar

# Show how to run instar
make run-instar

Prototypes:

# Build a specific prototype
make build-prototype PROTOTYPE=virtio-block5

# Build all prototypes
make build-all

# Build the shared guest-protocol crate
make guest-protocol

# Build devcontainer for a prototype
make build-prototype-devcontainer PROTOTYPE=virtio-block5

# Build the rust-lint Docker container
make build-lint-container

Cleaning:

# Clean a specific prototype's target directory
make clean-prototype PROTOTYPE=virtio-block5

# Clean all build directories (main + prototypes)
make clean-all

# Remove all devcontainer Docker images
make clean-devcontainers

# Remove the rust-lint Docker image
make clean-lint-container

# Remove everything (all targets + all containers)
make distclean

Linting:

# Run rustfmt and clippy checks
make lint

# Run with auto-fix
make lint-fix

# Install pre-commit hooks
make install-hooks

Integration Testing:

# Create Python venv for tests (testtools/stestr)
make test-venv

# Run safe integration tests
make test

# Run tests with verbose output (shows diffs)
make test-report

# Run all tests including malicious images (explicit opt-in)
make test-malicious

# Run tests inside container (as CI does)
make test-container

# Run split test targets (used by CI for parallel execution)
make test-container-core              # info, check, security, oslo-crossval
make test-container-convert-qcow2    # QCOW2/VMDK/RAW convert + compare
make test-container-convert-vhd      # VHD/VHDX convert (slowest)

# Clean test artifacts
make clean-tests

Fuzz Testing:

# Build a single coverage-guided fuzz target (uses the devcontainer)
make fuzz-build FUZZ_TARGET=fuzz_resize_planners

# Build every coverage-guided fuzz target
make fuzz-build

# Run a single target for a bounded wall-clock budget (seconds; default 60)
make fuzz-run FUZZ_TARGET=fuzz_resize_planners FUZZ_DURATION=300

# Run the seven snapshot shell harnesses (live differential
# verification against qemu-img; needs a built instar + /dev/kvm)
make snapshot-harnesses

See the "Coverage-Guided Fuzzing" section below for the target list and the nightly CI rotation.

Running:

# Show run instructions for a prototype
make run PROTOTYPE=virtio-block5

What the integration tests cover¶

The integration tests compare instar info output against qemu-img info to verify drop-in replacement compatibility, validate instar check against deliberately corrupt test images, cross-validate instar compare output against qemu-img compare, and cross-validate instar convert output against qemu-img convert. oslo.utils format_inspector cross-validation tests verify that instar's format detection, safety checks, and virtual size reporting agree with OpenStack's image safety gate. Adversarial image tests verify safe handling of compression bombs, circular/deep backing chains, integer overflow headers, boundary value edge cases (refcount order, oversized virtual sizes, VMDK grain sizes, VHDX dual headers, BAT beyond EOF), and format confusion attacks (polyglot files, truncated headers, VMDK descriptor attacks). CVE reproduction tests verify that 6 known qemu-img CVEs (CVE-2024-32498, CVE-2015-5163, CVE-2022-47951, CVE-2015-5162, CVE-2014-0223, CVE-2024-4467) are fully mitigated by instar's architecture. tests/test_snapshot.py adds 94 snapshot-subcommand tests: the 12-image list matrix against cross-version baselines, 12 JSON golden comparisons with a structural cross-check, mutation round-trips (create/delete/apply) with qemu-img check post-op assertions, error paths and qcow2-only enforcement, and empty-table behaviour. JSON goldens live in tests/golden/snapshot-list/. Test images are in the sibling instar-testdata/ repository.

See testing.md for the full test suite documentation, and testdata/README.md for the test image catalogue (benign, malicious, edge-case, and AFL-discovered images).

Directory structure¶

instar/
├── .devcontainer/  # Development containers
│   └── rust-lint/  # Stable Rust for linting
├── src/            # Main instar implementation
│   ├── vmm/        # Virtual machine monitor (host-side)
│   ├── core/       # Core guest initialization
│   ├── shared/     # Shared library code
│   ├── crates/     # Shared format parsing crates (no_std)
│   │   ├── qcow2/  # QCOW2 header, L1/L2, decompression, refcounts
│   │   ├── raw/    # MBR/GPT partition table detection
│   │   ├── vhd/    # VHD footer, dynamic header, BAT parsing
│   │   ├── vhdx/   # VHDX headers, region table, metadata, BAT, CRC-32C
│   │   ├── vmdk/   # VMDK4 header and descriptor parsing
│   │   ├── luks/   # LUKS header parsing, KDF, AFsplitter, decryption
│   │   ├── vdi/    # VDI header parsing, block-map lookup
│   │   ├── parallels/ # Parallels header parsing, BAT lookup
│   │   ├── qcow1/  # QCOW1 (v1) header, L1/L2 block-lookup
│   │   ├── dmg/    # DMG koly trailer, chunk-table, chunk lookup
│   │   └── ...     # Per-operation planner crates (measure, create,
│   │               # resize, rebase, commit, snapshot)
│   ├── operations/ # Pluggable operations (info, copy, check, compare, convert, measure, create, resize, rebase, commit, map, snapshot, amend, dd, bitmap, bench)
│   └── build.sh    # Build script
├── crates/         # Shared Rust crates
│   └── guest-protocol/ # Protocol Buffers messaging for guests
├── prototypes/     # Experimental implementations (reference)
│   ├── helloworld/     # Minimal KVM VMM with bare-metal guest
│   ├── helloworld2/    # Same, using rust-vmm vm-memory crate
│   ├── virtio-block/   # Virtio-block device emulation
│   ├── virtio-block2/  # With guest-protocol integration
│   ├── virtio-block3/  # With configurable sector sizes
│   ├── virtio-block4/  # With performance statistics
│   ├── virtio-block5/  # With ioeventfd optimization
│   ├── virtio-block6/  # With sparse/dynamic output support
│   ├── pluggable/      # Modular operations architecture
│   ├── pluggable2/     # Separate binary loading for operations
│   └── info/           # Image format detection (qemu-img info)
├── scripts/        # Build and check scripts
├── tests/          # Integration tests (Python/testtools)
│   ├── base.py         # Base test class
│   ├── manifest.json   # Test image definitions
│   ├── helpers/        # Test utilities
│   └── test_*.py       # Test files
├── docs/           # Design documents and research
│   ├── index.md    # Documentation index
│   ├── usage.md    # Platform usage analysis (oVirt, Proxmox, OpenStack)
│   ├── security.md # CVE analysis for image handling
│   ├── qcow2/      # QCOW2 format documentation
│   ├── vmdk/       # VMDK format documentation
│   └── raw/        # Raw format documentation
├── testdata/       # Test images for security validation
│   ├── benign/     # Safe test images (qcow2, raw, vmdk, vhdx, vpc)
│   ├── malicious/  # CVE exploit images (DANGEROUS)
│   └── downloaded/ # External test images (CirrOS, QEMU iotests, etc.)
├── Makefile        # Build and development automation
├── CHANGELOG.md    # Release history
├── SECURITY.md     # Vulnerability reporting and security policy
└── README.md

Releases¶

See CHANGELOG.md for release notes.

Release artifacts (pre-compiled Linux binaries) are published to GitHub Releases via the release workflow (.github/workflows/release.yml). Tags are signed with Sigstore. To cut a release:

make release VERSION=0.2.0
git push origin HEAD
git push origin v0.2.0

GitHub automation¶

This project uses Claude Code-powered GitHub automation for PR management.

Bot commands¶

Comment on a PR with these commands (requires write access):

Command	Description
`@shakenfist-bot please re-review`	Request a fresh automated code review
`@shakenfist-bot please retest`	Re-run functional tests without pushing a new commit
`@shakenfist-bot please attempt to fix`	Attempt to fix failing tests

GitHub issues¶

The automated reviewer creates GitHub issues for actionable items (fix/document). These issues are linked in the review comment with "Closes #N" syntax, so they're automatically closed when the PR merges.

Workflows¶

Automated Review: same-repository PRs automatically receive code review after CI passes, and GitHub issues are created for actionable items
Test Fixing: On-demand test failure resolution via PR comment

Pull requests from forks are not reviewed automatically. The reviewer runs Claude Code with --dangerously-skip-permissions on a runner holding a token with pull-requests: write, and the PR diff it reads is untrusted input, so a prompt injection in a fork's diff could reach a write-capable token. Fork contributions are reviewed by a human instead; asking a maintainer to push the branch to this repository will get it the automated review as well.

The reviewer itself is not defined here. automated_reviewer in .github/workflows/functional-tests.yml is a thin caller which names this project's test jobs in its needs: list -- the "CI passed" gate -- and delegates everything else to shakenfist/actions/.github/workflows/pr-auto-review.yml, which is shared across the Shaken Fist projects.

See .github/workflows/ for implementation details.

Self-hosted runners and Docker¶

Almost every job in this repository runs on the self-hosted runner pool ([self-hosted, debian-12, ...]), and those runners do not ship Docker. Since instar is built and tested inside the devcontainer image, any job that runs docker, make instar, make test-rust, make lint or any other container-backed Makefile target must install it first:

    env:
      DOCKER_BUILDKIT: 1

    steps:
      - name: Install Docker
        run: |
          sudo apt-get update
          sudo apt-get install -y docker.io
          sudo systemctl start docker
          sudo chmod 666 /var/run/docker.sock

Omitting the step does not fail at job start -- it fails part way through with docker: command not found, whenever the first container command is reached.

The one exception is mermaid-lint.yml, which runs on [self-hosted, vm, debian-12-docker, s]. That is the fleet image that ships docker.io, so it needs no install step -- but the label has to be listed in .github/actionlint.yaml or actionlint rejects the workflow.

Self-hosted runners and the GitHub CLI¶

The same applies to gh: the [self-hosted, debian-12, ...] runners do not ship it either, so any job that files an issue, opens a PR or otherwise calls the GitHub CLI must install it first. Do not paste an apt block into the workflow -- call the shared installer, which is a no-op when gh is already present:

      - name: Install GitHub CLI on host
        run: tools/ci/install-gh-cli.sh

The [self-hosted, claude-code] and [self-hosted, static] runners do have gh preinstalled, which is why the Claude Code workflows and the comment-triggered bot workflows call it directly. The static case is load-bearing rather than incidental: shakenfist/actions' pr-bot-trigger composite action -- the first step of every @shakenfist-bot please ... workflow across the fleet -- is built entirely on gh api and gh pr comment, so if those runners lacked gh no bot trigger anywhere would work. Instar's own pr-address-comments.yml run 24368074053 (2026-04-13) has that step passing on a static runner.

So the rule is per runner label, not per workflow: add the installer step on debian-12 (and any new pool that turns out to lack gh), and leave the claude-code and static jobs alone.

This one bites late rather than early. The gh call is usually the last thing a job does -- open the PR, file the issue -- so the job burns its full runtime, succeeds at the real work, and then dies on gh: command not found. That is exactly what happened to rust-nightly-bump on 2026-08-24: it validated nightly-2026-08-24, committed it and pushed the branch, then failed the run without opening the PR. Worse, in release.yml the gh call sits on a if: failure() error-reporting path, where a missing gh would have silently swallowed the report of the original failure.

Merge queue and the `develop` ruleset¶

develop is gated by a repository ruleset named "Develop branch" (not classic branch protection — the whole Shaken Fist fleet uses rulesets). It requires merges to go through GitHub's merge queue, which is what runs the seven-distro package matrix; see testing.md for what runs on a pull request versus in the queue.

The configuration is recorded here so it can be recreated if the repository ever is. It mirrors shakenfist/shakenfist's ruleset of the same name:

Setting	Value
Ruleset	"Develop branch", id `20783686` (created 2026-08-12)
Target	`refs/heads/develop`
Enforcement	active
Bypass	team `shakenfist/sf-can-skip-merge-queue`, mode `always`
Rules	`deletion`, `non_fast_forward`, `merge_queue`, `pull_request`, `required_status_checks`
Required checks	`Can enqueue` and `Can merge` (GitHub Actions, integration 15368)
Queue grouping	`ALLGREEN`, `max_entries_to_build: 1`, `max_entries_to_merge: 5`
Queue merge method	`MERGE`, min 1 entry, 5 minute wait
Check timeout	360 minutes
Required approvals	0 (`dismiss_stale_reviews_on_push: true`)

Can merge was added second, and the gap between the two was not theoretical. Can merge only runs on merge_group events, so until a real merge group had executed GitHub had never seen that check context — and requiring a context that has never reported blocks every merge, on a branch that had no protection to fall back to. The ruleset therefore shipped on 2026-08-12 requiring Can enqueue alone, and Can merge was added on 2026-08-15 once the queue had made the context exist.

In between, two PRs merged through the queue without the matrix gating them, because a job that is skipped reports success while a job that never runs reports nothing at all. Can enqueue carries an if test that excludes merge_group, so inside a merge group it skips, reports success, and satisfies the only required check. Both merges followed the same clock: Can enqueue skipped, and GitHub merged the entry thirty one seconds later with the seven-distro matrix still running. #496's matrix then took until 14:10Z to finish, and was green.

493's was not — Rocky 9 went red seventeen minutes after that PR had¶

already merged, Ubuntu 22.04 and Debian 12 followed, and the Can merge aggregate reported failure seventy five minutes post-merge. The regression it was reporting (the release image's glibc floor) sat on develop until #496 fixed it.

If you are recreating this repository, add both contexts up front only if you can also arrange for a merge group to have run; otherwise reproduce the two-step order above. To add a required context to a live ruleset, transform the exported object rather than PUTting the GET response back unchanged — the response carries fields that are not part of the update schema (id, _links, and "parameters": null on the rules that take no parameters):

gh api repos/shakenfist/instar/rulesets/20783686 | jq '{
  name, target, enforcement, conditions,
  bypass_actors: [.bypass_actors[] | {actor_id, actor_type, bypass_mode}],
  rules: [.rules[]
    | if .type == "required_status_checks"
      then .parameters.required_status_checks
             += [{context: "Can merge", integration_id: 15368}]
      else . end
    | if .parameters == null then {type: .type} else . end]}' > ruleset.json
gh api -X PUT repos/shakenfist/instar/rulesets/20783686 --input ruleset.json

Read it back and diff it against the intent afterwards; dropping bypass_actors from the payload silently removes the bypass.

Two other settings deserve explanation:

max_entries_to_build: 1 bounds the cost of the matrix. Only one merge group builds at a time, so the seven-wide fan-out is seven on-demand runners for one PR, not seven per queued PR.
The required checks are the two aggregate jobs, never the individual matrix entries. Entry names change whenever the distro list does, and a required check whose name no longer exists blocks every merge permanently. can_enqueue aggregates the pull-request jobs; can_merge aggregates the merge-queue jobs. Both use always() plus an event test so they always report, because a required check that never reports leaves the queue waiting forever.

To inspect or recreate it:

gh api repos/shakenfist/instar/rulesets --jq '.[] | "\(.id) \(.name) \(.enforcement)"'
gh api repos/shakenfist/instar/rulesets/<id>

.github/exported-config/ carries the nightly export of the live ruleset state, which is the machine-readable companion to the table above; the export proposes its updates as pull requests rather than committing directly.

Differential fuzzing¶

On-demand differential fuzzing compares instar against qemu-img on randomly generated images to find behavioral divergences:

# Run locally (requires instar binary and qemu-img)
python3 scripts/differential-fuzz.py \
    --instar src/target/release/instar \
    --iterations 100 \
    --seed 42

# Trigger via GitHub Actions (workflow_dispatch)
gh workflow run differential-fuzz.yml \
    -f iterations=1000 \
    -f seed=42

The fuzzer generates random images (varying format, size, cluster size, compression, data patterns), runs chains of operations (info, check, convert) against both tools, and reports divergences with full reproduction details.

When libyal tools are available (vmdkinfo, vhdiinfo, qcowinfo), the fuzzer also cross-checks instar output against these independent forensic-grade parsers. This provides a third opinion for QCOW2 (alongside qemu-img) and fills the gap for VMDK/VHD/VHDX where qemu-img check is unavailable.

See scripts/differential-fuzz.py for implementation details.

Coverage-guided fuzzing¶

Coverage-guided fuzzing uses cargo-fuzz (libFuzzer) to exercise the parser crates directly without the VMM/KVM stack:

# Inside the instar-build container:
cd src/fuzz
cargo fuzz run fuzz_qcow2_header -- -max_total_time=60

40 fuzz targets cover all parser crates (QCOW2, VMDK, VHD, VHDX, VDI, Parallels, QCOW1, DMG, RAW, LUKS) including header parsing, L1/L2 lookup, refcount traversal, and decompression, plus the create / resize / rebase / commit planners, the qcow2 check-repair planners (fuzz_check_repair), the map extent walkers, the snapshot table parser (fuzz_snapshot_parse), the snapshot refcount mutators (fuzz_snapshot_refcount), the dd window math (fuzz_dd_window), CHS geometry rounding (fuzz_chs_rounded_size), windowed read primitives (fuzz_dd_read), and the qcow2-write planner (fuzz_qcow2_write, which drives the write/copy-on-write planner through the crate's sim harness asserting the max_rc < 3 COW invariant oracle, and fuzz_qcow2_write_growth). Seed the corpus from instar-testdata:

python3 scripts/extract-fuzz-corpus.py --testdata /path/to/instar-testdata

The CI workflow runs nightly at 04:00 UTC. Crashes are minimized and filed as GitHub Issues with the security-audit label immediately. See src/fuzz/ for target implementations.

Build and dev containers¶

The build runs in two devcontainer images: a minimal debian:bullseye release build image (src/.devcontainer/build/Dockerfile, image instar-release) that produces the binary and packages at a low glibc floor, and the full Debian dev/test image (src/.devcontainer/Dockerfile, image instar-build) that runs the test, fuzz, and audit suites. make instar/deb/rpm use the former; everything else uses the latter. See docs/development.md for which target uses which image and why bullseye.

RPM dependency generation¶

make rpm runs cargo generate-rpm, and src/vmm/Cargo.toml sets auto-req = "auto" so the Requires list is derived from the built binary rather than hand-maintained. "auto" is not one implementation: cargo-generate-rpm uses /usr/lib/rpm/find-requires when that path exists, and otherwise silently falls back to a builtin parser of ldd -v output.

The fallback is broken and must never be taken. The builtin parser strips whitespace out of each ldd -v line, so a weak symbol-version reference — which glibc's ldd renders as libc.so.6 (GLIBC_2.25) [WEAK] — collapses into the dependency libc.so.6(GLIBC_2.25)[WEAK](64bit). No glibc package Provides that string, so the .rpm fails to install on every RPM distro with nothing provides libc.so.6(GLIBC_2.25)[WEAK](64bit). rpm's own elfdeps ignores VER_FLG_WEAK and emits the plain libc.so.6(GLIBC_2.25)(64bit), which resolves normally.

Both container images therefore install Debian's rpm package purely so /usr/lib/rpm/find-requires is present. It is not used to build the package. Do not remove it from either Dockerfile.

This is latent, not theoretical: it only bites when the toolchain starts marking glibc version needs weak. nightly-2026-07-22 emitted Flags: none for GLIBC_2.18/2.25/2.28/2.29/2.30; nightly-2026-08-17 emitted Flags: WEAK for the same five, which took out the Rocky 9, Rocky 10 and Fedora matrix jobs in the merge queue. .deb packaging is unaffected — cargo-deb uses dpkg-shlibdeps. To inspect what a built package actually requires:

rpm -qpR src/target/generate-rpm/instar-*.rpm
readelf -V src/target/release/instar   # Version needs section

Note that this class of breakage is only caught by the distro matrix, which runs in the merge queue rather than in the pull-request gate.

Toolchain pinning¶

Both devcontainer Dockerfiles pin the same Rust nightly via ARG RUST_NIGHTLY=nightly-YYYY-MM-DD — a broken floating nightly otherwise breaks every from-scratch image build (a 2026-07-24 nightly ICE'd compiling tokio inside cargo install cargo-audit and took out CI's "Build devcontainer" step). Renovate cannot bump rustup toolchain pins; instead the weekly rust-nightly-bump workflow (tools/ci/bump-rust-nightly.sh) rewrites and test-builds both images, then instar and the Rust test suite, against the newest published nightly and opens a bump PR only when everything passes. Do not un-pin the toolchain, and do not bump the pin by hand without at least building both images. (The lint container is separate and uses a stable rust: tag Renovate does manage; the dev image's Debian base is pinned by digest and Renovate walks it forward.)

Cargo tool pinning¶

The Rust nightly is not the only thing a from-scratch image build resolves at build time. Both Dockerfiles cargo install their tooling from crates.io — cargo-binutils, cargo-deb and cargo-generate-rpm in both images, plus cargo-fuzz and cargo-audit in the dev image — and those installs are pinned two ways:

An ARG <TOOL>_VERSION per crate, so the tool version itself does not float. Each ARG carries a # renovate: datasource=crate depName=<crate> comment directly above it, which a customManagers entry in renovate.json matches. Keep the comment adjacent to its ARG or the pin silently stops being managed. Renovate raises one PR per crate covering every file it appears in, so the dev and build images move together — never bump one file alone.
cargo install --locked, so each tool builds against the exact dependency versions its author released it with instead of re-resolving to the newest semver-compatible transitive dependencies.

tools/ci/check-devcontainer-pins.sh enforces both, from pre-commit and from the build-and-test CI job: it fails if the two images disagree about the nightly or about a shared tool version, if any cargo install line has lost its --locked, if a crate is installed at a hardcoded version rather than through its ARG, or if a pin's # renovate: comment is missing, detached, or names a different crate than the ARG. A quoted pin value fails too — renovate.json captures [^\s"]+, so ARG CARGO_DEB_VERSION="3.8.0" would match nothing and freeze silently.

Nothing in the guard is keyed off a list of tool names or a cargo- prefix. The shared set it compares across the two images is the intersection of what they actually install, so a tool added to both later is covered from the day it is added rather than when someone remembers to extend a list; and the Renovate-visibility rules are applied to whatever crates the install lines name, so a dependency whose crate is not called cargo-something gets the same treatment.

The guard also runs the customManagers regex as it is written in renovate.json today over both Dockerfiles, and requires it to find exactly the pins the guard found by its own parse — same crates, same versions, no more and no fewer (tools/ci/check-renovate-manager.py). Without that, the format rules above are only this repository's memory of what Renovate needs: a typo in matchStrings, or a managerFilePatterns entry that stops naming these files, would freeze all eight pins forever while the guard still reported them healthy. A frozen pin is worse than the floating install it replaced, because nothing errors anywhere.

The second is what protects against a broken transitive dependency, and it is not hypothetical: on 2026-09-03 tinyvec 1.13.0 was published with use alloc::vec::{self, Vec}, which shadows the vec! macro so the crate does not compile at all. cargo install cargo-audit picked it up within the hour and failed the "Build and test via devcontainer" step on every PR until upstream shipped 1.13.2. Nothing in the repository had changed. --locked would have held cargo-audit to the tinyvec its lockfile names.

Note the interaction with the Renovate rule that freezes src/.devcontainer/build/Dockerfile: that rule is scoped to the docker datasource, so it still pins debian:bullseye (whose glibc 2.31 sets the floor of the shipped binary) while leaving the crate pins in the same file managed.

CI tooling guards¶

The ci-tooling CI job runs the cheap guards over CI's own tooling: the test-partition check below, plus tools/ci/test-report-fuzz-crash.sh and tools/ci/test-pick-fuzz-artifact.sh for the coverage-fuzz helpers (see "Crash reporting" in docs/testing.md) and tools/ci/test-check-devcontainer-pins.sh for the devcontainer pin guard (see "Cargo tool pinning" above). It is also the job named in automated_reviewer's needs list, which is required to list every job that can fail a PR.

Integration tests are split across several CI jobs by stestr regex selectors (the test-container-* Makefile targets). tools/ci/check-test-partition.sh fails if any test_*.py test is run by no pull-request job. When you add a new integration test module or a new integration job, the guard validates that the new partition still covers everything; an orphan is a hard CI failure. Deliberate exclusions live in an allowlist in tools/ci/check-test-partition.py (currently just the malicious suite). See docs/testing.md.

How automated review works¶

The review job lives in the shared workflow shakenfist/actions/.github/workflows/pr-auto-review.yml, not in this repository. automated_reviewer in functional-tests.yml is only the caller: its needs: list names this project's test jobs, which is what gates the review on CI passing. The runner, the timeout, the bot-commit check and the fork restriction all live in the shared workflow.

Reviews run on same-repository pull requests only. The reviewer runs Claude Code with --dangerously-skip-permissions while holding a token with pull-requests: write and issues: write, and the PR diff is untrusted input, so a fork PR is skipped rather than reviewed. Fork PRs get a skipped job, not a failing one.

The automated reviewer outputs structured JSON. The shared workflow validates it, creates GitHub issues for actionable items (action=fix or action=document), and posts a human-readable rendering as a PR comment with the raw JSON embedded in a collapsed <details> section. The schema and the renderer live in shakenfist/actions alongside the reviewer.

The review comment includes links to the created issues with "Closes #N" syntax, so issues are automatically closed when the PR merges.

Each review item has an action field: - fix - Must be fixed before merging (creates an issue) - document - Documentation should be added (creates an issue) - consider - Optional improvement (reviewer suggestion) - none - Informational observation only

Workflow Files¶

.github/workflows/functional-tests.yml - Main CI, and the caller for the shared automated review workflow (shakenfist/actions/.github/workflows/pr-auto-review.yml)
.github/workflows/release.yml - Release workflow (Sigstore-signed tags, GitHub Releases with pre-compiled binaries)
.github/workflows/pr-re-review.yml - Manual re-review trigger
.github/workflows/pr-retest.yml - Manual retest trigger via bot command
.github/workflows/pr-fix-tests.yml - Test failure fixing
.github/workflows/test-drift-fix.yml - Scheduled/on-demand test maintenance
.github/workflows/differential-fuzz.yml - On-demand differential fuzzing (instar vs qemu-img + libyal)
.github/workflows/coverage-fuzz.yml - Coverage-guided fuzzing of parser crates (nightly + PR)
.github/workflows/rust-nightly-bump.yml - Weekly devcontainer Rust nightly pin bump (see "Toolchain pinning" above)
.github/workflows/codeql-analysis.yml - CodeQL static analysis (push/PR to develop, plus weekly cron)
.github/workflows/supply-chain.yml - gitleaks secret scanning on debian-13 (PR/push, plus weekly cron)
.github/workflows/agent-context.yml - skillsaw lint of AGENTS.md, CLAUDE.md and .claude/, by running the pre-commit hook (PR and push to develop, both path-filtered)
.github/workflows/lint.yml - rustfmt and clippy in the containerised toolchain (PR, path-filtered to the Rust tree)
.github/workflows/export-repo-config.yml - nightly export of this repository's GitHub settings for the fleet configuration audit
.github/workflows/renovate.yml - self-hosted Renovate dependency updates (hourly)
.github/workflows/mermaid-lint.yml - renders every mermaid diagram in the repository's markdown and fails on any that does not parse (PR, push to develop, and workflow_dispatch; path-filtered to markdown)

The self-hosted runners have no Docker preinstalled, so any job touching docker or a container-backed Makefile target needs an "Install Docker" step -- see "Self-hosted runners and Docker" in docs/development.md.

The debian-12 runners have no gh preinstalled either, so any job on those that calls the GitHub CLI needs a tools/ci/install-gh-cli.sh step -- see "Self-hosted runners and the GitHub CLI" in docs/development.md.

Scripts¶

scripts/differential-fuzz.py - Differential fuzzing script (instar vs qemu-img + libyal)
scripts/extract-fuzz-corpus.py - Seeds + restores the coverage-fuzz corpus from instar-testdata
tools/ci/install-gh-cli.sh - Installs the GitHub CLI on a self-hosted runner if absent (see "Self-hosted runners and the GitHub CLI" above)
tools/ci/fuzz-tier.sh - Computes tiered nightly per-target fuzz durations
tools/ci/report-fuzz-crash.sh - Files the security-audit issue for a coverage-fuzz crash (bounds the log excerpt, dedups against open issues; see "Crash reporting" in docs/testing.md)
tools/ci/pick-fuzz-artifact.sh - Chooses which libFuzzer artifact to report as the reproducer
tools/ci/test-report-fuzz-crash.sh, tools/ci/test-pick-fuzz-artifact.sh - Tests for those two; run them after any change (the ci-tooling CI job does)
tools/ci/check-glibc-floor.sh - Fails if the built instar binary needs a glibc above the published floor (GLIBC_2.31, Debian 11; see docs/installation.md). Runs immediately after make instar in both build-and-test and the release workflow. Do not raise the ceiling to make it pass: it means the release image's base moved, and src/.devcontainer/build/Dockerfile must stay on debian:bullseye
tools/ci/test-check-glibc-floor.sh - Tests for that check; the ci-tooling CI job runs it
tools/ci/claude-result.sh - Reads the JSONL stream a claude -p --output-format stream-json --verbose run leaves behind; --text reconstructs the assistant text (plus a diagnostic block when the run reported an error or the stream was truncated), --trailer emits the Assisted-By:/Co-Authored-By: pair naming the model the run actually resolved to. Called from test-drift-fix.yml
tools/ci/test-claude-result.sh - Tests for that helper; the ci-tooling CI job runs it
tools/mermaid-lint.sh - Renders every tracked markdown file that contains a mermaid fence, in the upstream mermaid-cli container (see "Diagrams" above); mermaid-lint.yml and the push audit's wave 1 both run it

📝 Report an issue with this page

Development¶

Building instar¶

Build and dev containers¶

Do not bump the release image's base¶

bullseye reached end of life: apt is pinned to a snapshot¶

The options that were not taken¶

Pre-commit hooks¶

Diagrams¶

Makefile¶

What the integration tests cover¶

Directory structure¶

Releases¶

GitHub automation¶

Bot commands¶

GitHub issues¶

Workflows¶

Self-hosted runners and Docker¶

Self-hosted runners and the GitHub CLI¶

Merge queue and the develop ruleset¶

493's was not — Rocky 9 went red seventeen minutes after that PR had¶

Differential fuzzing¶

Coverage-guided fuzzing¶

Build and dev containers¶

RPM dependency generation¶

Toolchain pinning¶

Cargo tool pinning¶

CI tooling guards¶

How automated review works¶

Workflow Files¶

Scripts¶

Merge queue and the `develop` ruleset¶