Skip to content

Integration Test Suite

The instar project includes a Python-based integration test suite that verifies instar info produces output identical to qemu-img info, that instar check correctly detects structural corruption in QCOW2 images, that instar compare produces byte-for-byte identical output to qemu-img compare, and that instar convert produces output identical to qemu-img convert. Since instar aims to be a drop-in replacement, any difference in output is considered a bug.

Architecture

The test suite uses:

  • testtools - Extended unittest framework with better assertions
  • testscenarios - Parameterized test scenarios
  • stestr - Parallel test runner with result storage

Tests compare instar output against either: 1. Live qemu-img output (for safe images - info, compare, convert) 2. Stored expected output files (for malicious images)

Test Categories

Safe Images (test_info_safe.py)

Tests against known-safe disk images. These run qemu-img directly and compare outputs character-for-character.

Malicious Images (test_info_malicious.py)

Tests against images designed to exploit vulnerabilities (e.g., backing file references to /etc/passwd). These use pre-stored expected output files instead of running qemu-img, since running qemu-img on malicious images defeats the security purpose of instar.

Check Validation Tests (test_check_formats.py)

Tests for the instar check operation: - Format detection: Verifies check correctly identifies QCOW2, VMDK, VHD formats - Per-format refusal: TestCheckVdiRefusal, TestCheckParallelsRefusal, TestCheckDmgRefusal, and TestCheckQedRefusal verify check refuses each read-only-input format with the correct exit code and message - Corrupt images: Tests against deliberately corrupt format headers (VMDK, VHDX, VHD) - QCOW2 structural validation: Uses 4 script-generated corrupt QCOW2 images: - Clean baseline (should pass with 0 errors) - Overlapping clusters (two L2 entries pointing to same host cluster) - Refcount-zero (referenced cluster with refcount=0) - Leaked cluster (refcount>0 but no L2 reference) - Unsafe quirks mode: Verifies non-QCOW2 formats are treated as raw with --unsafe-quirks

Corrupt QCOW2 test images are generated by instar-testdata/custom/check-validation/create-corrupt-images.py, which creates images with qemu-img/qemu-io and then surgically corrupts specific QCOW2 structures via binary manipulation.

Repair Tests (test_check_repair.py)

Tests for instar check --repair, codifying the phase-7-verified behaviour against the corrupt-fixture matrix (each on a tempdir copy, never the committed fixture):

  • Leaks/all tiers repair to clean: leaked + --repair=leaks, and refcount-zero / refcount-too-high / stale-copied + --repair=all, are asserted qemu-img check-clean afterward with the surviving data patterns still readable via qemu-io read -P.
  • Refuse paths stay byte-identical: corrupt-bit-set, snapshot-leak, and compressed-leak are sha256-identical after repair, with the snapshot and compressed data intact.
  • Overlapping is a safe partial repair: the genuine leak is reclaimed, the structural overlap remains, no new error classes appear, and instar exits 2 with repair-incomplete.
  • CLI: --repair+--chain rejection, clean-image no-op, qcow2-only not-supported on a raw image, and idempotence.
  • Repaired counters (TestRepairCounters): the per-class repaired-leaks / repaired-refcounts / repaired-corruptions counts reach the host over the CheckResultMessage protobuf and render in both the JSON and human --repair output, and are omitted on a read-only check so its schema is unchanged.

The structural tests assert post-repair state (qemu-img check-clean, data reads, byte-identity) rather than instar's exit code, since the exit code still reflects the detected corruption after a clean repair; the repaired-counter output is asserted separately by TestRepairCounters now that those counts travel on the guest→host wire.

Compare Tests (test_compare.py)

Tests for the instar compare operation, cross-validated against qemu-img compare:

Raw-vs-raw (TestCompareRawIdentical, TestCompareRawDifferent, TestCompareRawSizeMismatch, TestCompareRawJson): - Identical images: Self-compare and two identical files - Different content: Mismatch at offset 0 and at mid-file offsets - Size mismatch: Non-strict (zeros = identical), non-strict (non-zero = differs), and strict mode (always fails on size difference) - JSON output: Validates identical, first-mismatch-offset, total-bytes-compared, and size-mismatch fields

QCOW2-vs-raw (TestCompareQcow2VsRaw): - Identical content across formats (including all-zeros) - Different content reports correct mismatch offset - Cross-validated against qemu-img compare

QCOW2-vs-QCOW2 (TestCompareQcow2VsQcow2): - Identical and different content between two QCOW2 images - Virtual size mismatch handling - Cross-validated against qemu-img compare

Compressed QCOW2 (TestCompareQcow2Compressed): - Compressed QCOW2 vs raw with same content (zlib decompression) - Compressed vs uncompressed QCOW2 with same content - Cross-validated against qemu-img compare

Backing chains (TestCompareBackingChain): - QCOW2 overlay with raw backing file vs flattened raw (identical) - QCOW2 overlay vs different raw (mismatch detected) - Deep chain (3-level: top -> mid -> base) vs flattened raw (identical) - Two different QCOW2 backing chains with same virtual content (identical) - All scenarios cross-validated against qemu-img compare

Test images are created at runtime using qemu-img create, qemu-io write, qemu-img convert -c (for compressed), and qemu-img create -b (for backing chains), so no external testdata is needed.

qemu-img cross-validation: Every scenario verifies byte-for-byte identical stdout and matching exit codes with qemu-img compare.

Convert Tests (test_convert.py)

Tests for the instar convert operation (QCOW2 to raw), cross-validated against qemu-img convert. Per-format convert matrices cover VDI (TestConvertVdiToRaw), Parallels (TestConvertParallelsToRaw), QCOW1 (TestConvertQcow1ToRaw), and DMG (TestConvertDmgToRaw) input; QCOW1 also gets a dedicated smoke test module, tests/test_qcow1_smoke.py.

Basic conversion (TestConvertBasicQcow2ToRaw): - Empty QCOW2 image to raw - QCOW2 with written data to raw - Output size matches virtual size - All cross-validated against qemu-img convert

Compressed QCOW2 (TestConvertCompressed): - Compressed QCOW2 to raw (zlib decompression) - Compared against original raw source

Backing chains (TestConvertBackingChain): - QCOW2 overlay with raw backing flattened to raw - Deep chain (3-level: top -> mid -> base) flattened to raw - Cross-validated against qemu-img convert

Raw passthrough (TestConvertRawToRaw): - Raw to raw identity conversion

Error handling (TestConvertErrors): - Unsupported output format rejected - Nonexistent input file rejected

Manifest images (TestConvertManifestImages): - Converts real-world QCOW2 images from the test manifest - Cross-validates against qemu-img convert output - Skips images with cluster_size > 64KB (unsupported) - Skips images whose virtual_size exceeds available temp space

Resize Tests (test_resize.py)

Tests for the instar resize operation, structured around six surfaces totalling 114 tests:

Schema-drift tripwire (TestResizeBaselineMatrix.test_resize_cases_match_baselines): - Walks instar-testdata/expected-outputs/resize-info-json/<target>/<version>/ and asserts the on-disk case set matches the in-test RESIZE_CASES mirror. Catches drift between this mirror and the testdata generator.

Cross-version baseline matrix (TestResizeBaselineMatrix): - Per-(target, case) factory diffs instar create → instar resize output against the phase-10 qemu-img info JSON baseline. - ~22 active cases for qcow2 + raw; ~20 skipped (vmdk/vhd/vhdx where qemu rejects, -no-shrink rejection cases that surface 6 covers, and KNOWN_RESIZE_DIVERGENCES carry-forwards).

Live cross-validation (TestResizeCrossValidation): - 7 curated qcow2 + raw cases comparing instar end-to-end against the system qemu-img via instar info on both outputs.

Round-trip check (TestResizeRoundTripCheck): - instar create → resize → check for ~30 non-raw cases. - Catches resize-emitter regressions that produce files qemu-img info accepts but instar check flags.

Internal consistency for vmdk/vpc/vhdx (TestResizeConsistency): - 14 cases for the formats qemu can't resize, verified via instar info virtual-size match + instar check.

Targeted error paths (TestResizeErrorPaths): - 9 fixed tests pinning the host CLI rejection contracts: shrink-without-flag for raw + qcow2, subtractive-size without --shrink, invalid size strings, --preallocation=metadata on raw, --preallocation=falloc + --shrink, --object, --image-opts.

Wall clock: ~32 s serial / ~10 s with make test-container's --concurrency 4.

Snapshot Tests (test_snapshot.py)

Tests for the instar snapshot subcommand (phase 11 of PLAN-snapshot), 94 tests across five families:

  • List matrix (human): per baselined image, TZ=UTC instar snapshot -l byte-equals the host-resolved profile baseline from instar-testdata/expected-outputs/snapshot-list-human/ (80 qemu-img versions captured; instar tracks the modern ≥9.0 layout); plus the bare-filename-defaults-to-list check.
  • List goldens (JSON): --output=json byte-equals the instar-side self-baselines in tests/golden/snapshot-list/, plus a structural vmstate cross-check and a QMP-key schema test.
  • Mutation round-trips: create / delete / apply on tempdir copies with post-op qemu-img check clean, content verified via qemu-img compare, and structural behaviour on the name-collision / duplicate-name / cap-boundary fixtures.
  • Error paths: qcow2-only enforcement across raw / vmdk / vhdx, feature-gate refusals (zstd, dirty bit, external data file, LUKS), not-found exit codes, -U+mutating refusal, --image-opts rejection.
  • Empty table: empty stdout + exit 0; JSON emits [].

The suite is the CI regression net. The snapshot shell harnesses (tools/snapshot-{create,delete,apply}-{matrix, refusals}.sh and tools/snapshot-cli-parity.sh — seven scripts, 241 assertions) are the live differential layer: byte-identity against the host qemu-img from identical inputs, run with make snapshot-harnesses (requires a built instar and /dev/kvm; CI runs it in the functional-tests workflow's snapshot-harnesses job). They overlap by design.

Adversarial Image Tests (test_adversarial.py)

Tests verifying that instar safely handles malicious and malformed images without crashing, hanging, or consuming excessive resources. Uses the run_adversarial() helper in base.py which enforces timeouts (hang detection), memory limits via RLIMIT_AS (resource exhaustion), and signal checks (crash detection).

Phase 1 — CVE-adjacent attacks: - Compression bombs: Zlib and ZSTD compressed QCOW2 images with extreme expansion ratios. Verifies decompression buffer bounds are enforced and output files stay small. - Circular backing chains: 2-level cycle (A→B→A), 3-level cycle (A→B→C→A), and self-referencing (A→A). Verifies chain discovery detects the cycle and rejects it. - Deep backing chains: Chains at 16 levels (device limit) and 17 levels (exceeds limit). Verifies depth enforcement and correct rejection. - Integer overflow: L1 table size near u32::MAX, L1 size = 0, cluster_bits below minimum (8) and above maximum (22). Verifies checked arithmetic prevents undefined behavior.

Phase 2 — boundary value cases: - Refcount order edges: refcount_order = 7 (128-bit, invalid) and 255 (extreme value). Verifies clamping or rejection. - Oversized virtual size: 1 petabyte and u64::MAX virtual sizes. Verifies info reports the size and check doesn't allocate based on virtual size alone. - VMDK grain size: Zero and huge (2^63) grain sizes. Verifies checked_mul prevents division by zero and overflow. - VHDX conflicting headers: Dual headers with different sequence numbers and valid CRC-32C checksums. - BAT beyond EOF: VHD and VHDX images with BAT entries pointing past end of file. Verifies I/O error handling.

Phase 3 — format confusion: - Polyglot files: QCOW2 magic header with VMDK descriptor body, and QCOW2 magic with ELF binary content. Verifies format detection works (magic wins) and structural validation catches inconsistencies. - Truncated headers: QCOW2 v2 header cut at 32 bytes, VMDK with only 8 bytes (magic + version), VHD footer at 48 bytes. Verifies all operations fail gracefully with no crash. - VMDK descriptor attacks: Null bytes in descriptor, multiple extent declarations, and inflated 1MB descriptor size claim. Verifies parser handles adversarial text safely.

Test images are generated by scripts in instar-testdata/scripts/ (the private testdata repository). Scripts that generate adversarial or CVE-reproducer images must always be placed in instar-testdata, never in the public instar repository.

All generated images live in instar-testdata/custom/audit/.

Security Tests (test_security.py)

Tests verifying instar's security properties: - Backing file references are detected but not followed - External data file references are reported but not read - VMDK descriptor extent paths are not accessed

Running Tests

Make Targets

# Create Python virtual environment (first time only)
make test-venv

# Run safe tests (default, suitable for development)
make test

# Run CI-suitable tests
make test-ci

# Run all tests including malicious images (explicit opt-in)
make test-malicious

# Run with verbose output (useful for debugging diffs)
make test-report

# Clean test artifacts
make clean-tests

Direct stestr Usage

cd tests
source .venv/bin/activate

# Run all tests
stestr run

# Run specific test module
stestr run test_info_safe

# Run with verbose output
stestr run --serial -- --verbose

# List available tests
stestr list

Running against an installed package (distro matrix)

The make targets above run the suite against the in-tree build (src/target/release/instar). The distro-matrix CI instead runs it against the installed .deb/.rpm, inside a target-distro container, using that distro's own qemu-img as the differential oracle. tools/test-package-functional.sh is the per-distro runner:

# Full suite, against the .deb installed on Debian 12 (qemu-img 7.2.x)
tools/test-package-functional.sh src/target/debian/instar_*.deb debian:12

# Fast subset, against the .rpm installed on Rocky 9 (proves the dnf path)
tools/test-package-functional.sh --smoke \
    src/target/generate-rpm/instar-*.rpm rockylinux:9

# Dial concurrency down when several matrix containers share one KVM host
tools/test-package-functional.sh --concurrency 2 \
    src/target/debian/instar_*.deb ubuntu:22.04

# Replay one failure on one distro without paying for the whole suite
tools/test-package-functional.sh --select 'test_convert\.' \
    src/target/debian/instar_*.deb debian:13

To reproduce a CI matrix entry exactly as the merge queue runs it — same package resolution, same summary line — use the wrapper CI calls rather than the runner directly:

make package
PACKAGE_DIR=src/target/debian \
TESTDATA_PATH=../instar-testdata \
MATRIX_SELECT='test_map\.' \
    tools/ci/run-matrix-entry.sh 'Debian 12' debian:12 deb

MATRIX_SELECT exists for exactly this and is never set in CI: it makes the run partial, and a partial run must not be able to report as a green matrix entry. The wrapper prints a warning whenever it is set.

Classify before you attribute. Running two matrix containers at once — or any other heavy job on the host — starves the suite, and the resulting failures do not look like resource problems. The large test_convert re-encodes fail under load with Error: "convert operation failed" and Content mismatch at offset 0!, which reads as data corruption, and they abort early (~26s) rather than timing out at the ~110s they need when passing. Nine such failures appeared across the 2026-08-09 matrix inventory and every one passed on an idle host. A failure counts as a divergence only once it reproduces with the host otherwise quiet; --select gives you that replay cheaply. Same discipline as the differential-fuzzing spurious-divergence rule.

The design is tests from the source tree, binary from the package: the whole tests/ tree is copied into the container and driven by stestr, while the harness is pointed at the installed /usr/bin/instar via INSTAR_BINARY_PATH (so the packaged binary and its packaged guest binaries under /usr/lib/instar/ are what runs). The container runs as root (package + prerequisite install need it) and the repo is mounted read-only, so no .stestr/ artefacts leak into the host worktree. test_info_malicious and test_bench are excluded from the matrix run; --smoke restricts to a fast version/create/map/info subset for local one-distro checks.

TESTDATA_PATH (default ../instar-testdata) must already be git-LFS materialised — in CI that is tools/ci/prepare-testdata.sh; the script canary-checks for pointer files and refuses to run against them. This runner is distinct from tools/test-package-install.sh, which stays the fast packaging smoke check (file layout, --help, info/create/map).

Because this exercises instar against older qemu-img versions than the dev host's, it is the first thing to surface output-format parity gaps that single-version CI cannot: any command whose qemu-img output changed across versions, where instar hard-codes the newest form, will diverge here. See "qemu-img version profiles and the distro matrix" below.

Pull requests versus the merge queue

functional-tests.yml runs two different sets of jobs depending on the event, and the split is deliberate:

Event Jobs
pull_request ci-tooling, build-and-test, package-smoke, the three integration-* jobs, snapshot-harnesses, oslo-crossval-master, automated_reviewer, can_enqueue — all against the in-tree build
merge_group build-and-test, package-build, package-matrix (seven distros), can_merge
workflow_dispatch everything except ci-tooling and can_merge — this is how you dry-run the matrix without enqueuing anything. Note automated_reviewer also skips here, because it needs ci-tooling and a skipped dependency skips its dependents; the two aggregates use always() precisely so they report anyway

The merge queue does not re-run the PR integration jobs. That is a coverage argument rather than a cost one: each matrix entry runs the full Python suite against the installed package on its own distro, against that distro's qemu-img, so the queue's coverage is a superset of the PR jobs on seven distros instead of one. build-and-test is the deliberate exception — it stays in both as the cheap fast-fail, and package-build depends on it.

package-build builds one .deb and one .rpm and uploads them as a single artifact that all seven entries consume. This works only because the release binary is built on debian:bullseye (symbol floor GLIBC_2.30), so one artifact set installs everywhere down to Rocky 9's glibc 2.34. If a matrix entry fails while installing the package, that is a glibc-floor regression in the build image, not a test failure.

can_enqueue and can_merge are the two aggregate gates, and they are the names configured as required checks — never the individual jobs or matrix entries, whose names change whenever the distro list does. can_enqueue aggregates the pull-request jobs ("may this PR enter the queue?"); can_merge aggregates the merge-queue jobs ("is this merge group good?"). Both use always() plus an event test and a jq expression asserting every dependency ended success or skipped, because a required check that never reports leaves the queue waiting forever. See development.md for the ruleset itself.

Each entry writes a row to the job summary naming the distro, its live qemu-img version, and the test totals. The version is what distinguishes "instar broke" from "instar diverges at this qemu version boundary", so read it before attributing a red row.

Flake quarantine. Every matrix entry carries an explicit allow_failure: false. An entry that fails twice consecutively for an established environmental reason may be flipped to true with a linked issue, so one flaky distro cannot block every merge; the flag comes off when the issue closes. Note the sharp edge: continue-on-error makes the job report success to the needs context, so a quarantined entry stops gating merges entirely rather than merely tolerating its own failure. Never leave a flag behind after its issue is fixed.

Because merge_group does not inherit the pull_request trigger's paths: filter, this workflow always runs in the queue — including for docs-only changes. That is deliberate (a required check that never runs hangs the queue), at the cost of matrix latency on doc merges.

CI job layout and the partition guard

On a pull request the integration suite is split across several jobs (in .github/workflows/functional-tests.yml) so the long-running convert tests can run in parallel with everything else. Each job selects a subset with stestr regex filters defined in the Makefile:

Job Make target Selection
integration-core test-container-core everything except test_convert., test_compare., test_info_malicious. (the catch-all)
integration-convert-qcow2 test-container-convert-qcow2 test_convert./test_compare. minus Vhd
integration-convert-vhd test-container-convert-vhd test_convert.TestConvert.*Vhd
oslo-crossval-master (inline) test_oslo_crossval re-run against oslo.utils master

test_info_malicious.py is intentionally excluded from every PR job and runs only via make test-malicious.

Because integration-core is exclude-based, it silently absorbs any new test_*.py module — convenient, but it means a future refactor (e.g. turning it into an include-list like the convert jobs) could drop a whole module from CI with no test failure. The class-level convert split has a subtler gap: a test_convert class containing Vhd but not matching TestConvert.*Vhd would be excluded by the qcow2 job and missed by the vhd job.

The ci-tooling CI job guards against both. It runs tools/ci/check-test-partition.sh, which enumerates the suite with stestr list, reads the actual job selectors from the Makefile and workflow (no duplicated copy to drift), and fails if any test is run by zero jobs. The only hand-maintained input is the allowlist of intentional exclusions in tools/ci/check-test-partition.py (currently just the malicious suite); each entry carries a documented reason. Run it locally with:

tools/ci/check-test-partition.sh

Its own logic is unit-tested (stdlib only, no venv) via:

python3 -m unittest discover -s tools/ci -p 'test_*.py'

When you add a new integration job or change a test-container selector, the guard validates the new partition automatically; if it reports an orphan, either add the test to a job or (rarely) to the documented allowlist.

Test Image Manifest

Test images are defined in tests/manifest.json:

{
    "id": "cirros-qcow2",
    "path": "downloaded/cirros/cirros-0.6.3-x86_64-disk.img",
    "format": "qcow2",
    "safety": "safe",
    "run_in_ci": true,
    "description": "CirrOS minimal cloud image",
    "tags": ["qcow2", "cloud-image"]
}

Manifest Fields

Field Description
id Unique identifier for the test image
path Path relative to testdata root
format Expected disk format (qcow2, vmdk, vhd, etc.)
safety safe, caution, or malicious
run_in_ci Whether to include in CI test runs
unsafe_quirks_required If true, requires --unsafe-quirks flag for qemu-img compatibility
description Human-readable description
tags Searchable tags for filtering
expected_override Path to expected output file (for malicious images)

Unsafe Quirks Testing

Images marked with unsafe_quirks_required: true do not have valid format headers or partition tables. In default (secure) mode, instar rejects these files as "unknown format" rather than accepting them as raw images.

To test qemu-img compatibility for these images, use --unsafe-quirks:

# Default mode: rejects files without valid structure
instar info random-garbage.raw
# Error: Unknown format (no valid disk image header or partition table)

# Unsafe quirks mode: matches qemu-img behavior
instar info --unsafe-quirks random-garbage.raw
# file format: raw

See configuration.md and quirks.md for details on safe vs unsafe quirks.

Test Data Location

Test images are stored in a separate repository (instar-testdata) to keep the main repository small. The location is resolved in order:

  1. INSTAR_TESTDATA_PATH environment variable
  2. ../instar-testdata (sibling directory)

Binary fixtures are git-LFS backed

Every binary fixture in instar-testdata (*.qcow2, *.raw, *.vhd, *.vhdx, *.vmdk, *-backing, the bundled qemu-img binaries) is tracked with git-LFS. A checkout whose LFS objects have not been materialised leaves each fixture on disk as a ~131-byte pointer file, and the whole suite then fails with file format: unknown for every image — a failure that looks like a mass instar regression but is really a testdata problem. If you clone instar-testdata by hand, run git lfs pull after cloning; a working tree with real images (the fixture starts with its true magic, e.g. QFI\xfb for qcow2) rather than pointer text is required.

CI does this through tools/ci/prepare-testdata.sh — the single, LFS-aware testdata-prep helper shared by functional-tests.yml, coverage-fuzz.yml, and test-drift-fix.yml. It clones or updates the cached checkout, runs git lfs pull, and then guards against the pointer failure mode: it inspects known canary fixtures and hard-fails the job with an explicit "infrastructure problem, not a test regression" message if any is still a pointer, so a git-LFS outage can never be silently misattributed to instar (see GitHub issue #451 for the incident this guards against).

Expected Output Overrides

For malicious images where running qemu-img would be dangerous, store the expected output in tests/expected_outputs/:

tests/expected_outputs/
└── qcow2_backing_etc_passwd.txt

Reference this file in the manifest:

{
    "id": "qcow2-backing-passwd",
    "expected_override": "expected_outputs/qcow2_backing_etc_passwd.txt"
}

Image Notes

The docs/image_notes/ directory documents which test images exposed specific quirks or implementation details. When a test image reveals unexpected qemu-img behavior that requires compatibility work, create a markdown file documenting:

  • The specific values that revealed the behavior
  • How qemu-img handles the case
  • How instar now handles it
  • Links to relevant quirks documentation

See Image Notes for existing documentation.

Adding New Test Images

  1. Add the image to instar-testdata repository
  2. Add entry to tests/manifest.json
  3. For safe images: add scenario to test_info_safe.py
  4. For malicious images:
  5. Create expected output file in tests/expected_outputs/
  6. Add scenario to test_info_malicious.py
  7. If the image exposes new quirks: create docs/image_notes/<image-id>.md

Output Comparison

The test suite performs exact string comparison. On failure, it shows:

  • Unified diff with whitespace made visible
  • ␣ for trailing spaces
  • → for tabs
  • ↵ for trailing newlines
  • Raw repr() of both outputs for debugging

Before comparing, the harness normalises the fields that legitimately vary between runs rather than between qemu versions:

  • Disk size / actual-size is substituted from the live filesystem (st_blocks * 512, helpers/comparators.get_disk_size), because it reflects the current file's allocation, not a format decision. instar computes it the same way, so it always matches.
  • vmdk cid/parent-cid, the dirty flag, and vhdx log-size are stripped by assert_info_equivalent — a vmdk content-ID is a random nonce written at creation, not a stable output.

This matters for the version matrix: two adjacent baseline profiles often differ only in these normalised fields (see below), so a profile mismatch is invisible to the assertions until a genuinely version-gated field changes.

qemu-img version profiles and the distro matrix

instar emulates the qemu-img build installed on the host so its output is byte-identical to the real tool. Two mechanisms cooperate:

Runtime model (src/vmm/src/version.rs). instar runs qemu-img --version, parses major.minor.patch, and derives just two booleans: include_child_node (qemu ≥ 8.0 adds the Child node '/file' section) and include_dirty_flag (qemu ≥ 6.1 exposes the dirty flag). If qemu-img is absent it falls back to the newest profile. --qemu-version overrides detection.

Baseline profiles (instar-testdata). Because qemu changed some output within stable series, expected-outputs/<command>/version-map.json records several empirical profiles (e.g. profile-6-1-0, profile-7-2-19, profile-8-0-0, profile-8-1-0, profile-10-0-0, profile-10-2-0) and a version_to_profile map from full version strings to profile names.

Selecting a profile for the host qemu. base.py get_profile_for_installed_qemu / _pick_baseline_version_dir resolve the host version against the map with full major.minor.patch matching (_select_version_match): exact match, else the highest enumerated version ≤ the host within the same major.minor, else the highest version ≤ the host overall. A {major}.{minor}. prefix match (the previous behaviour) could not tell 7.2.0 (profile-6-1-0) from 7.2.22 (profile-7-2-19) and mis-selected on any 7.2.19+ host such as Debian 12. The single-qemu-version CI never exposed this; the distro matrix does.

Portable vs version-specific tests. test_info_safe and the --qemu-version baseline suites (create/resize/amend/commit/bitmap) drive instar with an explicit version per profile and compare to the stored baseline — they are qemu-version-independent and pass identically on every distro. The live-oracle suites (test_convert, test_compare, test_dd, test_check_*, test_map, test_measure, test_oslo_crossval) use the host's real qemu-img as the differential oracle and pick their comparison profile from the detected version, so they are the ones the matrix actually exercises.

The CI matrix (tools/probe-qemu-versions.sh). The qemu-img version each matrix distro ships, and the profile it selects:

Distro qemu-img --version Profile
Debian 12 (bookworm) 7.2.22 profile-7-2-19
Debian 13 (trixie) 10.0.11 profile-10-0-0
Ubuntu 22.04 (jammy) 6.2.0 profile-6-1-0
Ubuntu 24.04 (noble) 8.2.2 profile-8-0-0
Fedora latest 10.2.2 profile-10-2-0
Rocky/RHEL 9 10.1.0 profile-10-0-0
Rocky/RHEL 10 10.1.0 profile-10-0-0

Run tools/probe-qemu-versions.sh to refresh this table (it pulls each image and prints its qemu-img --version). Rocky 10 is pulled from the rockylinux/rockylinux org repo — the Docker Official rockylinux library stops at 9. Regression tests pinning the parser and the selection rule against these exact strings live in tests/test_version_detection.py and src/vmm/src/version.rs.

Profile selection is not the whole story. Debian 12's 7.2.22 is the only distro whose major.minor (7.2) transitions profile mid-series (at 7.2.19), so it is the only one the full-version fix re-points, and the profile-6-1-0 vs profile-7-2-19 baselines differ only in normalised fields (disk size, vmdk cid). Running the live-oracle suites against every distro's real qemu-img (via tools/test-package-functional.sh) then found divergences the baseline comparison never could — see below.

Known divergences (found 2026-08-09, fixed in phase 2b 2026-08-10). instar used to hard-code its newest-qemu behaviour for three things, so it diverged on every distro shipping an older qemu. All three are now version-gated:

Divergence Affected distros Boundary Status
map --output=json emitted compressed unconditionally Debian 12 (7.2.22), Ubuntu 22.04 (6.2.0) — 19 tests each 8.2.0 (absent at 8.1.5) Fixed: gated on include_map_compressed
map --output=json emitted present unconditionally none in the matrix (oldest is 6.2.0) 6.1.0 (absent at 6.0.1) Fixed: gated on include_map_present
snapshot -l used the newer column layout unconditionally Debian 12, Ubuntu 22.04, Ubuntu 24.04 (8.2.2) 9.0.0 (8.2.2 is old-form) Fixed: gated on snapshot_underscored_columns
Every VHD instar wrote declared a size its CHS geometry did not cover, so qemu < 10.0 read it short and truncated the tail Debian 12, Ubuntu 22.04, Ubuntu 24.04, RHEL 9 reader default changed at 10.0.0 Fixed: the writer now stamps creator app qem2

The third was not an output-format divergence at all but silent data loss in the VHD writer, found because the qcow1→vpc test compares a flattened round-trip. See docs/quirks.md ("VHD Virtual Size Calculation") and phase 2b's plan.

One divergence remains documented rather than fixed: instar applies the qemu 10.0+ VHD size rule to VHDs with an unrecognised creator app, where a pre-10.0 qemu-img would use the CHS product. It needs an image whose creator app is outside the known table and whose CHS disagrees with its disk_size, and no such image exists in the corpus. The rule is evaluated guest-side, so gating it would mean widening the guest ABI.

How the boundaries were measured. instar-testdata ships 80 static per-version qemu-img builds at qemu-img-binaries/x86_64/<version>/, covering 6.0.0 to 10.2.0. Run the real binary rather than reasoning from the version map or from qemu source — two of the three boundaries above were initially recorded wrongly from indirect evidence:

TD=../instar-testdata
$TD/qemu-img-binaries/x86_64/8.1.5/qemu-img map --output=json image.qcow2
$TD/qemu-img-binaries/x86_64/9.0.0/qemu-img snapshot -l image.qcow2

Both map and snapshot accept --qemu-version (as info already did), so both sides of every boundary are exercisable on the dev host rather than only inside the matrix.

Cross-profile baseline tests. The ordinary baseline tests compare only against the profile matching the installed qemu-img, so a single-version host checks exactly one side of every boundary — which is how the compressed divergence survived to the distro matrix. TestMapCrossProfile drives --qemu-version for every profile the version map declares and compares against that profile's own baselines, so a new boundary is caught wherever the suite runs. It immediately found one the matrix could not: map --output=json gained present in 6.1.0, and no matrix distro is old enough to notice.

qemu capability is not the same as qemu version. Distro qemu builds do not all carry the same block drivers, and the version-profile model says nothing about this. RHEL-family qemu-kvm (Rocky/RHEL 9 and 10) is built without the qed, qcow (qcow1), parallels, dmg, bochs and cloop drivers that Debian's qemu-utils carries — so on those distros qemu-img cannot act as the differential oracle for those formats at all. Tests that need such an oracle call skip_unless_qemu_supports(fmt) (tests/base.py), which probes the driver by opening a nonexistent file with an explicit format and looking for Unknown driver; tests that exercise instar alone (the check-refusal and adversarial suites) keep running there. Detect capability this way rather than from qemu-img --help, whose "Supported formats:" line qemu 10.x no longer prints.

A truncated run must never look green. subunit v2 refuses to write a packet larger than ~4MB, so a failing assertion that renders a multi-megabyte image buffer raises ValueError: Length too long and kills the stestr worker. Every test still queued on that worker silently never runs, and stestr exits 0 if nothing else failed — a partial run that reports as a pass (one Rocky run "passed" having executed 454 of 3253 tests). Two defences: compare image buffers with assert_bytes_identical (tests/base.py), which reports sizes and the first differing offset rather than the buffers, and tools/test-package-functional.sh, which fails the run outright on the crash marker, on any worker reporting N/A elapsed time, or on a full-suite test count far below the expected ~3250.

Environment Variables

Variable Description
INSTAR_TESTDATA_PATH Override default testdata location
INSTAR_BINARY_PATH Override default instar binary location

Differential Fuzzing

The project includes a differential fuzzer (scripts/differential-fuzz.py) that compares instar against qemu-img on randomly generated images. This is Phase 3 of the security audit plan (PLAN-audit.md).

How it works

For each iteration the fuzzer:

  1. Picks a random seed (logged for reproducibility).
  2. Generates a random disk image with qemu-img create, varying format (qcow2, raw, vmdk, vpc), virtual size (1M-1G), cluster size, compression, and data patterns (zeros, random, sparse, MBR).
  3. Creates separate copies for instar and qemu-img.
  4. Runs a random chain of 2-4 operations (info, check, convert, compressed convert, create, measure, resize, rebase, commit, map, snapshot) against both tools.
  5. Compares outputs at each stage: exit codes, normalized JSON info output, and converted file content (SHA-256 of raw-flattened output).

The op_snapshot arm (phase 13 of PLAN-snapshot) generates a fresh qcow2, then applies an identical random chain of snapshot -c / -d / -a and qemu-io write elements to the instar and qemu-img copies, asserting byte-identity of the whole image after every chain element (the qemu side runs with file.discard=ignore; dates and dead padding bytes are normalized per docs/quirks.md). Its first runs surfaced a real multibyte list-padding bug and delete's missing surviving-L2 COPIED refresh, both since fixed.

The op_repair arm (phase 10 of PLAN-check-repair) builds a fresh qcow2 with known data, injects one random qcow2 corruption (refcount-zero / too-high / leaked / stale-copied / overlapping), forks the corrupt file to two byte-identical copies, and repairs one with instar check --repair and the other with qemu-img check -r at a random tier. A three-tier oracle asserts: safety (unconditional — instar must never produce check-errors or raise the corruption/leak count above the original, even when it refuses); convergence (all tier only, where the two tools have matching scope — when instar claims a complete repair via repair-incomplete == false, the image must be qemu-img check-clean); and data equivalence (when both reach clean, the raw-flattened guest data must match). instar's deliberate refuse/partial behaviour is recorded as conservative, never a divergence; the leaks tier is excluded from convergence because it is intentionally narrower than qemu-img -r leaks (see quirks.md).

The op_commit, op_rebase and op_bench arms draw a snapshot flag (40% probability, phase 8 of PLAN-qcow2-write-infrastructure) and, when qemu-io is present, build a snapshot-bearing fixture that exercises the copy-on-write paths phase 7 opened — commit over a backing- or overlay-snapshot span, safe rebase of a snapshot-bearing overlay, and bench -w over a snapshot-shared span. For those fixtures the oracle gains the phase-7 read-back triple: active-view qemu-img compare identical + qemu-img check clean (with a refcount=1 reference=2 scan) + per-carrier snapshot read-back instar == qemu twin (via tests/helpers/snapshot_readback.py), the last of which the active-view compare alone cannot see. Non-snapshot fixtures keep their existing oracle. This folds in and retires the standalone scripts/cow-soak.py soak (phase 7e); a 300-iteration local soak ran 0 divergences.

Known quirks (see quirks.md) are excluded from comparison: disk size fields and format-specific metadata.

libyal cross-validation

When libyal tools are installed in the environment (libvmdk-utils, libvhdi-utils, libqcow-utils), the fuzzer adds two additional comparison layers:

  1. Info cross-check: Parsed fields from vmdkinfo, vhdiinfo, and qcowinfo (virtual size, format version, cluster size, etc.) are compared against instar's JSON output for the same image.
  2. Parse-success consistency: For each format, if the libyal tool successfully parses the image, instar check should report no errors (and vice versa). Disagreements are flagged as divergences.

This closes the gap where VMDK/VHD/VHDX had no differential reference for check validation (qemu-img check only supports QCOW2), and provides a third independent opinion for QCOW2. libyal tools are optional — the fuzzer degrades gracefully when they are unavailable.

Running locally

python3 scripts/differential-fuzz.py \
    --instar src/target/release/instar \
    --iterations 100 \
    --seed 42 \
    --log-dir ./fuzz-logs

CI integration

The fuzzer runs automatically via .github/workflows/differential-fuzz.yml at three tiers:

Trigger Iterations When
pull_request 100 PR changes fuzzer script or workflow
push to develop 200 Post-merge smoke test
schedule 1000 Nightly at 02:00 UTC
workflow_dispatch configurable Manual trigger

On failure, the workflow uploads logs as artifacts and auto-files GitHub Issues with the security-audit label, including the seed, iteration, image attributes, and a reproduction command.

Reproducing a divergence

Each divergence report includes the seed and iteration number. To reproduce:

python3 scripts/differential-fuzz.py \
    --instar src/target/release/instar \
    --iterations <ITERATION + 1> \
    --seed <SEED> \
    --fail-fast

Phase 6: Coverage-Guided Fuzzing

Coverage-guided fuzzing uses cargo-fuzz (libFuzzer) to exercise the no_std parser crates directly, without the full VMM/KVM stack. A mock CallTable backed by fuzzer input provides the I/O layer, allowing libFuzzer to explore malformed input space that differential fuzzing (Phase 3) cannot reach.

Fuzz targets

40 targets across the parser and planner crates, organized in src/fuzz/:

Target Crate Type
fuzz_format_detect shared Buffer-based
fuzz_qcow2_header qcow2 Buffer-based
fuzz_qcow2_l1l2 qcow2 CallTable
fuzz_qcow2_refcount qcow2 CallTable
fuzz_qcow2_decompress qcow2 CallTable
fuzz_vmdk_header vmdk Buffer-based
fuzz_vmdk_grain vmdk CallTable
fuzz_vhd_footer vhd Buffer-based
fuzz_vhd_bat vhd CallTable
fuzz_vhdx_header vhdx Buffer-based
fuzz_vhdx_metadata vhdx CallTable
fuzz_raw_partition raw Buffer-based
fuzz_luks_header luks Buffer-based
fuzz_vdi_header vdi Buffer-based
fuzz_vdi_bat vdi CallTable
fuzz_parallels_header parallels Buffer-based
fuzz_parallels_bat parallels CallTable
fuzz_qcow1_header qcow1 Buffer-based
fuzz_qcow1_table qcow1 CallTable
fuzz_dmg_table dmg CallTable
fuzz_dmg_chunk dmg CallTable
fuzz_measure_calc measure Buffer-based
fuzz_measure_scan all parsers CallTable
fuzz_create_emitters create Buffer-based
fuzz_resize_planners resize Buffer-based
fuzz_rebase_planners rebase Buffer-based
fuzz_commit_planners commit Buffer-based
fuzz_amend_planners amend Buffer-based
fuzz_map_iter all parsers CallTable
fuzz_snapshot_parse qcow2 CallTable
fuzz_snapshot_refcount snapshot Buffer-based
fuzz_check_repair check Buffer-based
fuzz_dd_window dd Buffer-based
fuzz_chs_rounded_size dd Buffer-based
fuzz_dd_read dd CallTable
fuzz_bitmap_parse qcow2 CallTable
fuzz_bitmap_planners bitmap Buffer-based
fuzz_bench_schedule bench Buffer-based
fuzz_qcow2_write qcow2-write Sim harness
fuzz_qcow2_write_growth qcow2-write Buffer-based

Buffer-based targets call parser functions that take &[u8] directly (e.g. QcowHeader::parse(data)). CallTable targets use the mock CallTable from src/fuzz/src/lib.rs to simulate sector-based I/O from the fuzzer input. Sim harness targets (only fuzz_qcow2_write) drive a crate's own Vec-backed simulation harness — here crates/qcow2-write's sim module, feature-gated #[cfg(any(test, feature = "sim"))] and OFF in the production build.

The two snapshot targets (phase 12 of PLAN-snapshot) cover the streaming snapshot-table parser (fuzz_snapshot_parse drives for_each_snapshot_entry and the planner converter against adversarial qcow2 fragments) and the mutator primitives (fuzz_snapshot_refcount dispatches one of seven ops per exec — refcount increment/decrement/swap, the COPIED-flag walker, the contiguous allocator, the flag helpers, and the table-serialisation round-trip — asserting semantic invariants such as precheck-never-mutates, inc/dec byte-identity, and allocator containment).

fuzz_check_repair (phase 9 of PLAN-check-repair) dispatches one of four crates/check repair planners per exec — leak reclamation, count accumulation, refcount correction, and COPIED reconciliation — asserting sub-byte-masked containment (co-resident refcount entries in a shared byte are preserved at widths 1/2/4), raised/lowered/freed tally correctness, the overflow→AmbiguousCorruption and bounds→MisalignedAccess error classifications, idempotence, and the "correct generalises reclaim" cross-check. The COPIED walker's deep invariants are delegated to fuzz_snapshot_refcount.

The two crates/qcow2-write targets (phase 8 of PLAN-qcow2-write-infrastructure) fuzz the write planner rather than a parser. fuzz_qcow2_write decodes a fixture archetype (clean / backing-present / shared-data / shared-L2 nested / owned-L2 / zero-flag-target) at a cluster size {512, 4 KiB, 64 KiB, 2 MiB} and runs a bounded plan_write / plan_flush sequence through the sim harness, asserting the copy-on-write invariant oracle after every operation: max_rc < 3 (the COW corruption signature — the single most important invariant), snapshot-shared clusters byte-preserved and never freed, no dangling / past-EOF L1/L2 pointer, and — after a flush — OFLAG_COPIED set iff refcount is exactly 1. A WriteError refusal is a valid outcome, not a crash. fuzz_qcow2_write_growth feeds geometry to the growth module's plan_refcount_growth, asserting no overflow, the self-coverage invariant, and cap adherence. Both are registered in the fast tier (tools/ci/fuzz-tier.sh); their bring-up shake-out found no planner bug. (Note: plan_refcount_growth has a debug-only non-convergence debug_assert! guard that fires only in debug builds on out-of-envelope petabyte-scale geometry; in release it returns GrowthOverflow gracefully, and production only passes bounded geometry — softening that guard to always return overflow is a recorded follow-up, not a bug.)

Running locally

# Inside the instar-build container:
cd src/fuzz

# Run a single target for 60 seconds
cargo fuzz run fuzz_qcow2_header -- -max_total_time=60

# Run with specific corpus
cargo fuzz run fuzz_qcow2_header corpus/fuzz_qcow2_header/

# Minimize a crash
cargo fuzz tmin fuzz_qcow2_header artifacts/fuzz_qcow2_header/<crash>

# Generate coverage report
cargo fuzz coverage fuzz_qcow2_header

Corpus seeding

The seed corpus is extracted from instar-testdata using:

python3 scripts/extract-fuzz-corpus.py --testdata /path/to/instar-testdata

This copies test images into per-target corpus directories under src/fuzz/corpus/, filtered by format. Header-only targets receive truncated copies. Hand-crafted minimal valid inputs are also generated for each format.

It additionally restores the accumulated corpus pushed by prior nightly runs. Each nightly run pushes coverage-increasing inputs to instar-testdata/custom/fuzz-corpus/<target>/; on the next run those entries are copied straight back into corpus/<target>/ by target name (restore_pushed_corpus()), so coverage compounds across runs. This matters most for the targets whose inputs are not recognizable image formats — the window-math, CHS-geometry and planner fuzzers — which would otherwise re-seed cold every night because format-based routing cannot place their entries. Entries are content-addressed, so restoration is idempotent.

CI integration

The CI workflow (.github/workflows/coverage-fuzz.yml) runs: - Nightly at 04:00 UTC, all targets, with tiered per-target durations (see below). - PR validation: single-target smoke test when fuzz/parser code changes. - Post-merge (push to develop): 15s per target. - Manual dispatch: configurable duration and target selection.

Tiered nightly durations

The nightly run has a fixed wall-clock budget (450 min, inside the 480 min job timeout). Rather than splitting it evenly, the run plan is computed by tools/ci/fuzz-tier.sh: the fast-saturating targets (pure window math, CHS rounding, and the planner/emitter crates) take a short fixed slice (300s — they reach steady coverage in well under a minute), and the deep parser/format targets split the remainder. With the current 27 targets that gives the 17 deep targets ~24 min each versus ~17 min under an even split.

This is one of two levers for keeping per-target time useful as the target count grows; the other is corpus persistence (see Corpus seeding). When the deep-tier share computed by fuzz-tier.sh falls to the fast-tier floor (~300s), stop cutting time and shard the targets across multiple CI jobs instead. Sharding only adds real throughput if the self-hosted runner pool has spare physical cores during the nightly window, since each libFuzzer target pins a core — confirm core availability before adding jobs.

Crash reporting

Crashes are minimized with cargo fuzz tmin and filed as GitHub Issues with the security-audit label immediately when found, by tools/ci/report-fuzz-crash.sh. New corpus entries are pushed to instar-testdata/custom/fuzz-corpus/ after nightly runs, and restored by target name on the next run so coverage compounds.

These rules make that reporting safe, most of them learned from a month of silently broken nightlies:

  • The log excerpt is bounded in bytes, and never travels on a command line. cargo fuzz prints the failing input as a std::fmt::Debug dump on a single line, so with -max_len=4194304 one crash can produce a 370KB log line. The excerpt is therefore clipped per line (cut -b), then per line count, then per byte, scrubbed to valid UTF-8, and handed to jq with --rawfile. The original code passed a tail -30 through jq --arg, and an 81KB crash input in fuzz_rebase_planners made that a 371KB argv entry — over Linux's 128KiB MAX_ARG_STRLEN, so jq exited with "Argument list too long".
  • A reporting failure is never fatal to the run. The fuzz step runs under bash -e, so that jq failure aborted the whole step at the first crash: 21 of the 40 targets were never fuzzed, no issue was filed, and the corpus push was skipped — every night from 2026-07-16 to 2026-08-11. Reporting failures are now counted and warned about per target, and the loop continues to the remaining targets. The failure is raised by a final Fail on unreported crashes step that runs with if: always() after the corpus push and the log upload, so an unreportable crash still turns the run red without costing the night's corpus — and without hiding the artifacts that are the only remaining way for that crash to reach a human.
  • Anything read out of a fuzz log is treated as hostile bytes. Both the excerpt and the signature are NUL-stripped, length-bounded and re-encoded with iconv -c, because a panic message can itself carry fuzz data and jq — like the GitHub API — rejects invalid UTF-8. Log scraping uses grep -a: without it grep decides a log containing raw mutated bytes is binary, prints nothing, and every such crash silently degrades to a signature of unknown crash.
  • The excerpt is windowed on the crash, not on the end of the log. A real cargo fuzz run failure prints the panic, then ~30 stack frames, the SUMMARY, the artifact path, the Debug dump and cargo-fuzz's reproduction block. In the 379KB log that motivated this change the panic is line 30 of 91, so a tail -n 30 window starts at line 62 and contains no panic line and no panic message at all. The window is five lines before the first panicked at/SUMMARY: and 25 after, trimmed from the end so the panic survives the byte cap; with no such line to anchor on — a build failure, a truncated log — it falls back to the tail.
  • The same crash is not refiled every night. Since the run no longer stops at the first crash, a recurring crash would otherwise file one issue per target per night, and fuzz-autofix.yml only drains one issue per day. An open security-audit issue whose title matches the target and whose body carries the same dedup_key gets a comment instead of a duplicate. A lookup that fails falls through to filing, because a duplicate issue is a much smaller problem than an unreported crash; --no-dedup forces that behaviour by hand.

The signature is the first panicked at/SUMMARY: line and the line after it: Rust prints the location and the message separately, and panicked at fuzz_rebase_planners.rs:278:17 on its own does not identify a crash.

Dedup matches on dedup_key rather than on that signature, because a panic message interpolates the fuzz-derived values that provoked it — the real crash behind this change reads Write patch 0 (72057594037927944..72057594037928200) exceeds total_file_size (281076066929798). Two inputs hitting one assertion produce two different signatures, so exact matching would file a fresh issue every night for a single bug. The key is the location plus the message with standalone numbers collapsed to N. Digits inside identifiers are left alone, so qcow2 and qcow3 do not merge: a duplicate issue is only noise, whereas two different bugs sharing one issue loses a crash. For the same reason the file:line:col is never normalized — two assertion sites in one file stay two issues — but the thread id in thread '<unnamed>' (47) panicked at … is dropped, since it varies run to run.

Not every fuzz failure is a Rust panic. An OOM, a timeout or a deadly signal anchors on libFuzzer's SUMMARY: line instead, and the line after that is the MS: mutation line, which ends in a per-input base unit: <hex>. Hex survives digit collapsing, so for a SUMMARY-anchored failure the key is the SUMMARY: line alone; including the MS: line would give every recurring OOM a new key and a new issue every night. Issues filed before dedup_key existed are still matched on their signature.

Run the reporter by hand against a downloaded coverage-fuzz-logs artifact to check what an issue would say, without filing anything:

tools/ci/report-fuzz-crash.sh fuzz_rebase_planners \
    src/fuzz/artifacts/fuzz_rebase_planners/crash-<hash> \
    coverage-fuzz-logs/fuzz_rebase_planners.log --dry-run

tools/ci/test-report-fuzz-crash.sh exercises the reporter against synthetic logs — the 370KB single line, raw mutated bytes, a missing log, a missing crash file, and the dedup decisions with a stubbed gh — and asserts the emitted body still satisfies the field predicate fuzz-autofix.yml validates against. Its fixture reproduces the layout of a real libFuzzer log, not just its content, because an earlier ten-line fixture let a tail-anchored excerpt look correct while on a real log it captured 28 stack frames and no panic.

Both suites run on pull requests in the ci-tooling job, and need only bash and jq:

tools/ci/test-report-fuzz-crash.sh
tools/ci/test-pick-fuzz-artifact.sh

Choosing which artifact to report

tools/ci/pick-fuzz-artifact.sh decides which file in src/fuzz/artifacts/<target>/ is the reproducer. It used to be inline YAML in the workflow, which is where two of the three bugs behind the broken month were hiding, so it is a script with tests now.

Do not pass -max_len to cargo fuzz tmin: it supplies its own, and a second one trips libFuzzer's assert(MaxInputLen == 0), so minimization fails and leaves a 0-byte minimized-from-* artifact behind — which an unsorted find | head -1 would then happily report as the reproducer for a later crash. The picker therefore skips minimized-from-* when choosing what to minimize, and afterwards is asked separately for the non-empty minimized-from-* that tmin produced; without that second step the issue would quote the size of, and the reproducer point at, the original large artifact, and minimization would cost CI time without improving anything.

The empty-file filter applies only to minimized-from-*, where an empty file means tmin failed. A 0-byte crash-* is a real crash — libFuzzer writes one when a target panics on the empty input — and filtering it out would send the workflow down its no-artifact branch, filing nothing and telling the reader to go and check the build.

Artifacts are taken in an explicit order of preference — crash-, oom-, leak-, timeout-, slow-unit-, then anything else — rather than by sorting the names. Lexicographic order happens to put crashes first, but it also puts slow-unit- ahead of timeout-, and libFuzzer writes a slow-unit- file for any input over 10s (its default -report_slow_units). Since these targets run against 4MB inputs, a slow unit can easily be sitting in the directory when a real timeout arrives, and the reported reproducer would then be the wrong file.

Automated bug fixes

The CI workflow (.github/workflows/fuzz-autofix.yml) runs daily at 06:00 UTC and picks up open security-audit issues. It invokes Claude Code (30-turn limit) to diagnose and fix the crash, then verifies the fix by rebuilding and running core tests. Two attempts per issue; failed issues are labelled autofix-failed for human attention. Complexity guardrails prevent runaway fixes (max 3 files, no cross-crate changes, no new dependencies).

📝 Report an issue with this page