Skip to content

VHD and VHDX differencing output

Status: Complete

Prompt

Before responding to questions or discussion points in this document, explore the instar codebase thoroughly. Read relevant source files, understand existing patterns (VMM structure, guest operation layout, shared crate conventions, call table ABI, format parsing, test infrastructure), and ground your answers in what the code actually does today. Do not speculate about the codebase when you could read it instead. Where a question touches on external concepts (QCOW2, VMDK, VHD/VHDX, LUKS, KVM, virtio, disk image formats), research as needed to give a confident answer. Flag any uncertainty explicitly rather than guessing.

All planning documents should go into docs/plans/.

Consult ARCHITECTURE.md for the overall system structure (host VMM, KVM guest, call table, device emulation). Consult AGENTS.md for build commands, project conventions, code organisation, and the security model summary. Consult docs/ for format-specific documentation (docs/qcow2/, docs/raw/, etc.) and docs/commentary/ for architectural decisions and design rationale.

This plan carries the mandatory push audit phase described in PLAN-TEMPLATE.md, and the Merged column that phase needs. The column is filled in as each phase lands, not reconstructed afterwards.

Situation

The 2026 technical goal VHD/VMDK writers done has four leaves: VHD fixed output (shipped), VHD/VHDX differencing output, VMDK twoGbMaxExtent* output, and vmdk/vpc/vhdx preallocation. This plan takes the differencing leaf. It is the only one of the three outstanding leaves that needs no call-table change, which is why it goes first: multi-extent VMDK output is blocked on multi-output-device support in the call table, and preallocation is blocked on a per-format BAT population pattern.

What exists today, measured on 2026-09-05 against the binary built from d59cc40 and qemu-img 10.0.11:

  • instar create -f vpc -b parent.raw -F raw child.vhd 16M fails with "create failed: invalid option for target format". plan_vhd (src/crates/create/src/lib.rs:767) and plan_vhdx (:919) both return CreateError::BackingFileUnsupported when a backing reference is present, each with a comment deferring the work as "too complex for phase 1" of PLAN-create.md.
  • src/crates/vhd/src/lib.rs knows DISK_TYPE_DIFFERENCING = 4 (:93) and VhdState::init accepts it (:578), but nothing in the crate parses the dynamic header's parent fields or its eight parent locator entries.
  • src/crates/vhdx/src/lib.rs finds the parent locator metadata item and then discards it: parent_loc_offset and found_parent_loc are assigned and immediately let _ = ... at :517-519. Only the HasParent file-parameter bit survives, as VhdxMetadata::has_parent (:462), and VhdxState::init rejects any image that sets it (:842).
  • Reading a differencing VHD therefore returns the child's allocated blocks and zeros everywhere else, with no diagnostic. instar convert -O raw on the vhd-differencing.vhd fixture exits 0 and writes a 10 MiB raw image composed as if the parent did not exist. instar info reports no parent. map is the only op that refuses (src/operations/map/src/main.rs:459-462), and check refuses only the VHDX case (:1555).
  • CHANGELOG.md:1922 claims VHD input support for "fixed, dynamic, differencing with backing chains". The chain half of that claim has never been true.

What the outside world does:

  • qemu-img creates neither. qemu-img create -f vpc -b base.vhd -F vpc child.vhd 16M and the vhdx equivalent both fail with "Backing file not supported for file format 'vpc'" / "'vhdx'". qemu's vpc and vhdx drivers have no differencing write path at any shipped version.
  • qemu-img reads a differencing VHD the same way instar does: qemu-img info on the fixture reports a plain 10 MiB vpc image and never mentions a parent. So instar's silent read is qemu-parity, but both are silently wrong rather than right.
  • This makes differencing output an instar-only capability with no qemu-img oracle for creation or composition, like the vmdk/vhd/vhdx resize and vmdk rebase divergences already recorded as note 8 in docs/format-coverage.md. Every other write path instar has shipped was validated against qemu-img. This one cannot be, so the plan has to establish an oracle before it writes anything. The absence is not total, and the plan should not overstate it: step 1b MEASURED that qemu-img 10.0.11 validates the VHD head footer's checksum and refuses a corrupted one ("Could not open: Incorrect header checksum"), so that one field is externally cross-checked even though qemu never resolves the parent. The dynamic header checksum, the tail footer copy's checksum and the whole locator table are not.
  • Debian 13 packages libvhdi-utils (vhdiinfo, vhdimount) and python3-libvhdi from the libyal project, which does implement VHD and VHDX parent chains. Phase 1 accepted it as this plan's oracle; open question 2 carries the verdict and the two gaps that come with it -- there is no vhdiexport, and libvhdi never parses the VHD parent locator table.

The fixtures are not usable as they stand. instar-testdata/custom/format-coverage/vhd-differencing.vhd has disk_type = 4 in its footer but its parent unique id, parent unicode name and all eight parent locator entries are zero: it is a type marker, not a differencing disk, and its companion vhd-diff-base.vhd is not referenced by it. There is no VHDX differencing fixture at all.

The generator is not missing, though — phase 2's survey found it in the other repository from where the master plan looked for it. scripts/create-vhd-testdata.sh in instar produces both existing VHD fixtures, writing into ../instar-testdata/custom/format-coverage by default, and its comment at :150-152 states the strategy outright: qemu-img cannot create differencing VHDs, so it patches disk_type from 3 to 4 in a dynamic image and recomputes the checksums. The type marker was deliberate, and the manifest entry (tests/manifest.json:197-206) describes it honestly as "patched from dynamic for type acceptance testing". Two tests depend on it (tests/test_check_formats.py:1877, :1900), so phase 2 supplements rather than replaces it.

The house pattern this reveals — generator in instar, binaries in instar-testdata, registration in instar's tests/manifest.json — means phase 2 spans both repositories and lands as two pull requests, and its Merged cell carries a record from each.

Related planned work lives in the testdata repository: instar-testdata/docs/plans/PLAN-extra-coverage.md priority 7 proposes five adversarial parent-locator fixtures (absolute /etc/passwd, ../../../etc/passwd, UNC, and eight mutually disagreeing locators). That priority is unstarted, and it becomes directly relevant the moment instar parses a locator table.

The host is further along than the rest of this picture suggests, which phase 1's survey established and which matters most to phase 11. discover_backing_chain (src/vmm/src/main.rs:2519) is not qcow2-only: it walks a chain with circular-reference detection, a depth limit and a path allowlist, and it already carries a non-qcow2 special case in the VMDK flat-descriptor short-circuit that resolves parentFileNameHint. Its security knobs, security.backing_path_allowlist and security.max_chain_depth, are declared at src/vmm/src/config.rs:65 and :67. Composition extends that function; it does not write one.

Step 1c narrowed the read-side defect, and the narrowing matters to phase 4. The silent misread is VHD only. On a differencing VHDX every read op already fails, because VhdxState::init rejects a parent at the crate level before any op-specific logic runs -- so convert, compare, dd, bench and measure inherit a refusal none of them wrote. What they inherit is a generic message rather than a diagnosis: instar convert -O raw on a Hyper-V differencing VHDX gives "convert operation failed", and instar compare given the same differencing VHDX as both arguments reports "Content mismatch at offset 0!" -- a file differing from itself. Phase 4's job is therefore two-sided: turn the VHD silence into a typed refusal, and turn the VHDX generic failures into the same typed refusal, rather than assuming VHDX is already correct because it exits non-zero. Both measured 2026-09-05 against the d59cc40 binary.

Two issues now record the read-side defects: #547 for the VHD silent misread, and #548 for compare reporting a content mismatch between a differencing VHDX and itself. Phase 4 closes both.

Mission and problem statement

Make instar create -f vpc|vhdx -b PARENT -F FMT child produce a differencing disk that an independent implementation resolves against its parent, and stop instar reading differencing children as though they had none.

In scope:

  • Differencing VHD output: disk_type = 4, parent unique id, parent timestamp, parent unicode name, and a parent locator table carrying the single entry open question 3 settles on, with both checksums correct and the BAT wholly unallocated.
  • Differencing VHDX output: the HasParent file-parameter bit, a populated parent locator metadata item carrying the parent_linkage GUID and the locator path entries.
  • Parent-locator parsing in crates/vhd and crates/vhdx, to the standard the rest of the format crates hold: no_std, panic-free, every offset and length from the image bounds-checked before use.
  • A defensible read-side answer for differencing children, so that no op silently composes the wrong image: a refusal first, and then real chain composition, so instar ends the plan able to read back what it writes.
  • An external oracle, and fixtures generated by a script that lives in the testdata repository rather than by hand.

Out of scope, and deliberately left to their own work:

  • Multi-extent VMDK output and vmdk/vpc/vhdx preallocation, the other two leaves of the same goal.
  • resize of a differencing image, which docs/resize.md:215-216 already defers pending the parent-locator update path this plan builds.
  • VHDX log replay, which remains rejected as it is today.

Open questions

  1. Read side: refuse, or compose? RESOLVED 2026-09-05 by the operator: both, in that order. The plan refuses first and composes last. Phase 4 turns the silent parent-ignoring read into a typed refusal, which closes the wrong-data hole immediately and holds while the emitters land; phases 11 to 17 then implement real chain composition for VHD and VHDX, the way instar already composes qcow2 backing chains -- the host attaches each chain member as its own virtio device and the guest takes an input_device_count. The refusal is therefore an interim state inside this plan rather than its endpoint, and instar finishes the plan able to read back everything it writes. Phase 4 is still worth its own phase: it is the only part of the read-side answer that has to be true before the emitters ship, and it is a defect fix rather than a feature.
  2. Is libvhdi a sufficient oracle? RESOLVED 2026-09-05 by step 1a: yes, with one named gap. libvhdi 20240509 (Debian libvhdi-utils, libvhdi1 and python3-libvhdi, all 20240509-2+b1) resolved Hyper-V produced differencing chains for both formats -- fat-differential.vhd, ntfs-differential.vhd and their .vhdx counterparts from the log2timeline/dfvfs test corpus, creator application win, images nothing in this project wrote -- reporting a parent identifier equal to the parent's own identifier and the correct parent filename. Its compositions matched the content the chains were built to represent byte for byte: cmp exit 0 for our generated VHD and VHDX chains against their intended raw images, with parent-only and child-only controls differing, and a sector-provenance analysis of the composed Hyper-V VHD chain finding every composed sector attributable to exactly one file. Two corrections to how the oracle is driven, both from step 1a: vhdiexport does not exist -- libvhdi ships vhdiinfo and vhdimount, and Debian builds the latter without FUSE -- so composition runs through the python3-libvhdi binding with an explicit set_parent() rather than a CLI export; and libvhdi never parses the VHD parent locator table, since libvhdi_parent_locator* is reached only from the VHDX metadata path and VHD resolution uses the parent unicode name field alone. That table therefore has no content oracle at all, which is the single biggest gap here: phase 5 must not assume otherwise and phase 8's assertions on it are structural only. Step 1b then added one field back that libvhdi does not cover: qemu-img 10.0.11 does validate the VHD head footer's checksum and refuses a corrupted image with "Could not open: Incorrect header checksum", where libvhdi opens it and resolves the parent regardless. The dynamic header checksum has no oracle in either tool. The full verdict, tool versions, command lines and the list of fields nothing external checks are in the phase 1 plan's Result -- step 1a and What has no oracle sections.
  3. Which locator entries do we emit? RESOLVED 2026-09-05 by step 1b: one entry, in slot 1, whose platform code describes the string the user actually gave us -- W2ru when the typed backing path is relative, W2ku when it is absolute -- with slots 2 through 8 left zero. This is a deliberate divergence from Hyper-V. MEASURED in both corpus VHDs, Hyper-V writes exactly two populated entries: W2ku carrying the absolute path in slot 1 at offset 1088 and W2ru carrying the child-relative path in slot 2 at 1112, with slots 3 to 8 zero. What decides against copying that is a host-side fact: run_create_nonraw resolves the typed backing path against the output's directory for opening the file but sends the guest typed_backing.as_bytes() unchanged (src/vmm/src/main.rs:16913-16963). The guest therefore has one string, and writing two entries would mean fabricating the other -- .\<basename> is simply wrong whenever parent and child are in different directories, and a fabricated locator is worse than an absent one. Passing a second, host-resolved path would need a new call table field, and this plan's premise that differencing output needs no ABI change (see phase 7's rationale below) is worth more than a cosmetic match to Hyper-V. The oracle cannot arbitrate: libvhdi ignores the VHD locator table entirely, so it cannot tell us whether one entry is enough for other readers, and no Windows host is in this plan's reach to ask. The falsifier is a Windows or Hyper-V rejection of a single-entry child; adding a second entry later is additive to the emitter and to nothing else. The VHDX side follows the same rule for the same reason: parent_linkage plus exactly one of relative_path or absolute_win32_path, and never volume_path (which needs a Windows volume GUID) or parent_linkage2. Reopened in part, 2026-09-15, by the review of the phase 5 pull request (#568). This answer settled which entries to emit; it did not consider that the one path string the guest is handed is a POSIX path, while W2ku and W2ru are defined as Windows paths and measured Hyper-V output writes .\fat-parent.vhd and C:\Projects\.... The emitter as built writes the typed path verbatim under those codes, so a Windows reader could resolve it drive-relative. Issue #570 carries the options and the evidence, and must be settled before phase 7 removes the guard that currently stops any user reaching the emitter. The reasoning above — one entry, never a fabricated second — is unaffected, as is the parent unicode name field, which is the only one libvhdi and qemu read.
  4. Does an instar-only capability need an opt-in flag? RESOLVED 2026-09-05: no flag. instar already performs vmdk/vhd/vhdx resize and vmdk rebase where qemu-img refuses on every shipped version, unflagged, and records them as note 8 in docs/format-coverage.md. Differencing output is a recorded divergence in the same way rather than a gated one.
  5. Parent path resolution and its security posture. RESOLVED 2026-09-05 by phase 1's survey, and the resolution is a description of code that already exists rather than a new rule to write. discover_backing_chain (src/vmm/src/main.rs:2519) is not qcow2-only: it already performs circular-reference detection, depth limiting and allowlist checking for every chain the host walks, governed by security.backing_path_allowlist and security.max_chain_depth (src/vmm/src/config.rs:65 and :67), and it already carries a non-qcow2 special case in the VMDK flat-descriptor short-circuit. Differencing parents go through that same function -- phase 11 extends it rather than writing one -- so a path read out of an image is resolved relative to the child's directory and checked against the allowlist before anything opens it, and nothing in the guest ever opens a path read out of an image. Phase 3 is still required to be bounds-check-clean against PLAN-extra-coverage priority 7 inputs; that requirement is unchanged by this resolution.
  6. Do we pull in the adversarial fixtures now? RESOLVED 2026-09-05: yes, in phase 2, taking priority 7 of instar-testdata/docs/plans/PLAN-extra-coverage.md (the absolute /etc/passwd, ../../../etc/passwd, UNC and eight-mutually-disagreeing-locator cases). Phase 3 is the code that needs them, and generating the happy-path and adversarial sets from one script is cheaper than two passes.
  7. Must a differencing child's parent share its format? RESOLVED 2026-09-05 by decision 5 of the phase 1 plan: yes, with a typed error otherwise. Hyper-V requires VHD parents for VHD children and VHDX for VHDX, and emitting a chain no implementation can resolve is worse than a typed refusal. instar's create -b path currently accepts any detectable parent format; phase 7 wires the error.

Execution

Each phase gets its own detailed plan file before implementation begins; this table is the tracking source of truth. The Merged column records what put each phase on develop -- the merge commit of its pull request, or a first..last range for a phase that landed directly -- because the push-audit phase runs PUSH-AUDIT.md over the union of those ranges, and git diff develop...HEAD is empty once the phases have landed. A phase that lands in instar-testdata records instar-testdata <sha> (#pr) and is audited there.

Phase Plan Status Merged
1. Semantics pin, oracle selection, and the doc correction PLAN-differencing-phase-01-pin.md Complete 8b81a0f (#549)
2. Real differencing fixtures, happy-path and adversarial (instar + instar-testdata) PLAN-differencing-phase-02-fixtures.md Complete instar-testdata 77f5f589f0 + 623a30866f + 3eed61bf75 (direct to main); instar 1a677c77 (#552)
3. Parent-locator parsing in crates/vhd and crates/vhdx PLAN-differencing-phase-03-parse.md Complete 42e879f (#558)
4. Read-side policy: close the silent parent-ignoring read PLAN-differencing-phase-04-read-policy.md Complete f981374 (#563)
5. plan_vhd differencing emitter PLAN-differencing-phase-05-vhd-emitter.md Complete 9a80776 (#568)
6. plan_vhdx differencing emitter PLAN-differencing-phase-06-vhdx-emitter.md Complete 882d098 (#577)
7. Guest create op and host CLI wiring PLAN-differencing-phase-07-guest-host.md Complete 99d7d24 (#581)
8. Rust unit tests and Python integration tests PLAN-differencing-phase-08-tests.md Complete c416abd (#588)
9. Coverage fuzzing of the locator parsers PLAN-differencing-phase-09-fuzz.md Complete 044ad77 (#594)
10. Documentation PLAN-differencing-phase-10-docs.md Complete 036252f (#598)
11. Composition: host chain discovery and info --chain PLAN-differencing-phase-11-chain-host.md Complete c66f8b2 (#603)
12. Composition: guest VHD sector-bitmap read path PLAN-differencing-phase-12-vhd-compose.md Complete b4ae7fc (#615)
13. Composition: guest VHDX sector-bitmap read path PLAN-differencing-phase-13-vhdx-compose.md Complete d677823c (#624)
14. Composition: rollout across the chain-walker operations PLAN-differencing-phase-14-op-rollout.md Complete 7fef1321 (#630)
15. Consistency for the non-composing operations PLAN-differencing-phase-15-op-consistency.md Complete f14554cf (#646)
16. Composition: integration tests and fuzz PLAN-differencing-phase-16-compose-tests.md Complete d7a4973f (#652)
17. Composition: documentation PLAN-differencing-phase-17-compose-docs.md Complete 60e4381c (#659)
18. Push audit: PUSH-AUDIT.md over every phase above PLAN-differencing-phase-18-push-audit.md Complete

Sequencing rationale

Phase 1 comes first because it is the only phase that can invalidate the rest: if no oracle exists, the shape of phases 8 and 9 changes and the operator should know before any emitter is written. It also answers open questions 1, 2, 3 and 7, and its first deliverable has already landed -- commit a93615d on this branch corrected docs/create.md, which claimed vpc and vhdx honoured backing_file when both planners reject it.

Phase 2 precedes phase 3 because a parser with no real input is a parser with no test. Phase 3 precedes both emitters because parse-then-emit lets each emitter be checked by instar's own reader before the external oracle is involved, which is how every other format crate in this repository was built.

Phase 4 sits before the emitters deliberately. It is the phase that fixes an existing defect rather than adding a feature, and putting it first means the tree is never in a state where instar writes differencing disks while still silently misreading them.

Phases 5 and 6 are independent of each other and could be parallelised; VHD goes first because its parent locator table is the simpler structure and the lessons carry into VHDX.

Phase 6 carries a trap step 1b found by reading the tree, and it is worth stating here because it changes what phase 8 can test. The VHDX parent_linkage key is the parent's DataWriteGuid, settled by measurement against Hyper-V bytes and confirmed against SPEC(VHDX) 2.6.2.6.3 and libvhdi's source. But vhdx::build_header derives the DataWriteGuid from the sequence number alone (src/crates/vhdx/src/lib.rs:2201, the two GUID writes at :2206-2215), and every instar VHDX writer passes sequence numbers 1 and 2 -- plan_vhdx at src/crates/create/src/lib.rs:1161 and :1163 and the convert op at src/operations/convert/src/main.rs:4469 and :4492. Every VHDX instar has ever written therefore shares one active-header DataWriteGuid, which makes libvhdi's parent-identity check vacuous for instar-written chains: any instar parent satisfies any instar child. plan_vhd has the same hole for the same reason -- it writes UUID_ZERO as the footer unique id of every image it creates (src/crates/create/src/lib.rs:958, :1021 and :1072), and that field is exactly what a differencing child copies into its dynamic header at offset 552. Phase 5 did not change that: the child's copy comes from opts.parent_unique_id (:977), which the create operation still fills with zeros until phase 7. Phase 6 should confirm the VHDX half against a real instar-produced image (the claim is read from code, not measured on output) and consider giving created images a real DataWriteGuid; phase 8's negative identity test must be built against a third-party parent either way, because an instar-created parent cannot fail it. Discharged by phase 7, which built exactly that test: tests/test_create.py round-trips both formats against third-party fixtures and asserts the fixture identity is non-zero before comparing, so the comparison is not zeros against zeros.

Phase 8 carried one debt from phase 5, now discharged and recorded here because the phase 5 plan is not where phase 8's planner will look. create -f vpc -b is still refused at the guest by an explicit guard that phase 7 removes, and that refusal had no committed test: the guard is in the guest binary and crates/create's harness cannot reach it, so phase 5 verified it by hand against a built binary. Phase 6's review round added test_create_vhd_and_vhdx_reject_backing to tests/test_create.py, covering vpc and vhdx together, so phase 8's brief no longer needs to and phase 7 gets a failing test if it removes either guard too early.

Phases 8 and 16 both build fixtures with partially populated blocks, and both must account for a libvhdi defect step 1a found and did not fix: libvhdi_block_descriptor_read_sector_bitmap_data decodes the VHD per-block sector bitmap with an unmasked shift, so once any higher bit in a bitmap byte is set, every later sector covered by that byte reads as present in the child. It was measured on Hyper-V's own fat-differential.vhd and reproduced deterministically with a probe that predicted seven wrong sectors and got exactly those seven. VHD fixtures must therefore keep parent-owned and child-owned sectors out of the same bitmap byte, or carry expected output that accounts for the bug; discovering it in phase 16 instead would look exactly like an instar bug. The VHDX branch of the same function is correct, and instar create output is unaffected because a freshly created child has a wholly unallocated BAT and no sector bitmaps at all.

Phases 11 to 17 are the composition work, and they come last because they are the only part that can be built on everything else: composition needs the locator parsing from phase 3 to find a parent, the emitters from phases 5 and 6 to generate chains to read, and the fixtures from phase 2 to read chains instar did not write. Phase 14 replaces phase 4's refusal op by op, so each op moves from "refuses, correctly" to "composes, correctly" and never passes back through "silently wrong".

Composition is a plan's worth of work on its own -- it is the half of this plan that touches the guest read path, where the output half touches only the writer -- so it is decomposed here rather than left as one phase to be split later, the way PLAN-resize-followup-01 split its own:

  • Phase 11, host side. Resolving a differencing parent to a real file, applying the same resolution and depth rules qcow2 backing chains get, so that info --chain walks a VHD or VHDX chain -- the cheapest possible proof the host half works, and it needs no guest change at all. Narrowed by phase 11's survey on 2026-09-28, which found two-thirds of this bullet already built. info --chain exists and already annotates a differencing child with the parent it declined to follow (run_info, src/vmm/src/main.rs:10269), and chain-member device attachment is format-agnostic already (open_chain_devices at :2831 and write_chain_device_entries at :2765 dispatch on nothing, and device format codes for Vhd and Vhdx are allocated -- docs/chain-config.md:63). What is actually left is the seven-line VHD/VHDX gate at :2701, one resolution rule per format, and that command's output.
  • Phases 12 and 13, guest side. The two formats are not the same problem and each is a self-contained read-path change, so they are separate phases that can be planned, reviewed and reverted independently. VHD first, for the same reason its emitter goes first. Narrowed by phase 12's survey on 2026-09-30, which found the guest chain walker already built and already carrying a VHD arm: read_chain_virtual_cluster (src/crates/qcow2/src/lib.rs:7930) dispatches per format at :8486, ChainStates already holds vhd_states (:9385) -- every line number in this bullet is as it stood before phase 12, which added about 2,800 lines to that file; the two a phase 13 reader follows are corrected at the end of the bullet -- and an unallocated BAT entry already descends to the next device -- which is the correct reading for a differencing child's wholly-parent-owned block. What is missing is the allocated-block case, and the same function's extended-L2 subcluster path (:8106-8130) already demonstrates the shape: coalesce runs from a bitmap and recurse into the backing chain for the sub-ranges this device does not own. The phase 4 refusal these phases were expected to relax is a single if in init_chain_states (:9528 for VHD, :9556 for VHDX), not a guard inside either parser crate. Post-phase-12 locations, for phase 13: the VHDX refusal is src/crates/qcow2/src/lib.rs:11298, and the VHDX arm of read_chain_virtual_cluster is :10269. Falsified by phase 12's execution: the planned condition -- lift the refusal when another device follows the child -- is unsafe, because init_chain_states's device_count counts a flat, possibly multi-chain device array (compare packs two chains into one), not the length of the child's own chain. Phase 12 built the composing arm but left the refusal unconditional; the guard moved into the reader, which already knows its own chain's bounds. What relaxing the refusal per operation actually needs is filed as issue #614, and it is a phase 14 precondition, not solved by phases 12 or 13 alone.

  • Phase 14, rollout across the chain-walker operations. Turning the refusals into composition for the operations that already read through the guest chain walker: convert, compare, bench, rebase, and dd by inheritance. Rescoped by phase 14's survey on 2026-10-05, which falsified most of what this bullet said. It is not "mechanical", and the seven operations it named were the wrong seven:

  • rebase reads through init_chain_states (src/operations/rebase/src/main.rs:1201) and was not named. Lifting the refusal changes it whether the phase plans for it or not, so it is in scope.

  • dd is not a separate operation. run_dd (src/vmm/src/main.rs:14629) ends in execute_convert, so it needs the host half and no guest work.
  • map and measure hold no ChainConfig at all and check only validates members independently, so for those three "composition" means building chain support rather than lifting a refusal. They are phase 15.
  • The refusal is four independent families, not the single if the phase 12 bullet promised: init_chain_states serves convert, compare, bench and rebase, while measure (:422, :441), check (:1593, :1996) and map (:463, :531) each refuse in their own code.
  • The host gate is per caller kind, not per operation. src/vmm/src/main.rs:3047 refuses to resolve a differencing parent for every ChainUse::Compose caller, and the comment justifying it rests on "nothing in instar can compose a VHD or VHDX chain yet" -- a premise phase 14 makes false for five operations and leaves true for three.

Needs issue #614 settled first: per-operation composition means lifting init_chain_states's refusal, and phase 12 found that doing so safely requires knowing a device's own chain boundary, which device_count alone does not give it.

It also needs issue #623 settled first, and for the same kind of reason. Phase 13 found that a differencing VHDX pads its BAT out to whole chunk groups where a dynamic one does not, and that the writer's calculate_bat_layout sizes the region by the shorter dynamic rule. The phase 13 reader reaches the padded entries but caps itself at the declared region, deliberately, so that the differencing images that load today keep loading. The consequence is that lifting the refusal before #623 lands ships a reader which cannot resolve the sector bitmaps of images instar itself wrote whenever that region is sized exactly -- a silent read failure on our own output, which is the worst shape this gap could take. Treat both issues as gates on phase 14's first step rather than as work the phase can absorb. Phase 14's survey added one correction here too: #623 reaches five consumers, not just the writer -- create twice, measure, resize and convert -- so fixing it changes measure's predicted size and resize's target layout as well as the bytes instar writes.

Phase 14 also takes issue #625, the VHDX walker accepting a block offset that overlaps the BAT or metadata region. Phase 13 filed it and left it, because while every read entry point refuses a differencing source the path is reachable only from crate tests. Phase 14 is the change that makes it reachable from a user command on an untrusted image, so it is the phase that should close it.

  • Phase 15, consistency for the non-composing operations. Rescoped by its own survey, and no longer the chain-support phase this section first described. What holds from that description: map and measure have no chain notion at all -- neither host discovers a chain and neither guest op consumes a ChainConfig -- and check's validate_chain walks a chain's members to validate each independently rather than to compose a read.

What does not hold is that phase 15 should build the composition. Measured against qemu-img 10.0.13, qemu-img check does not walk the chain at all, so check needs no composition to reach parity; while qemu-img map and qemu-img measure both do compose, for every format, which makes their gap qcow2-chain parity work rather than anything differencing introduced. map has refused a backing pointer since eb6e23f, before this plan began, as phase 4's survey already recorded. Both were deferred by PLAN-map.md and PLAN-measure.md, which are now Status: Complete, so the "which plan owns it" question this section posed had no answer: neither candidate owner is open. They leave this plan as issues #641 and #642, to be taken together by whichever plan picks them up, since both need multi-device plumbing in an op that has none.

commit's disposition was settled by phase 14 rather than left to phase 15: run_commit passes DifferencingComposition::Unsupported because its guest op ignores the ancestor slots the host fills in, so resolving a differencing parent could only turn a working command into a path error. It is correct and untested, and phase 15 supplies the test. The src/vmm/src/main.rs:7265 reference above has drifted and now points inside run_rebase.

What phase 15 does own is the boundary phase 14 left behind. The composing readers now refuse a VHDX payload block overlapping a declared region (#625) and map and measure do not apply that test, so the three operations disagree about what a valid block is (#634); check and bench attach a chain without the MAX_CHAIN_DEVICES guard the other five have (#633); and the comments justifying the check and measure refusals say instar cannot read a differencing image, which five operations now do. Those refusals stay -- these two report on one image, so a clean verdict or an allocation figure would present a partial view as a whole one -- with the lift question filed as #643 rather than settled by a phase scoped to consistency.

  • Phase 16, tests and fuzz. Cross-validation against the phase 1 oracle for chains instar wrote and chains it did not, plus coverage fuzzing of the compose path. Phase 13 left it one specific gap to close: every multi-group and partial-group VHDX compose test runs at 512-byte logical sectors, where chunk_ratio is 4096. At 4096-byte sectors it is 32768, so the padded BAT bound and the chunk-group stride are arithmetic nothing drives at the larger geometry. Its harness drives the python3-libvhdi binding with an explicit set_parent(), and its VHD fixtures are subject to the sector-bitmap caveat above. Phase 9 fuzzes the locator parsers; composing a chain is new surface, and a malicious child pointing at a well-formed parent is a different input space from a malformed locator table.

Three corrections from phase 16's survey on 2026-10-11. The binding is the right route but not for the reason stated: this bullet originally contrasted it with "a CLI export", and there is no such tool to contrast it with. Debian 13's libvhdi-utils 20240509-2+b1 ships vhdiinfo and vhdimount and nothing else -- measured by extracting the package. The real alternative is a FUSE mount. python3-libvhdi is a separate package and is not in the devcontainer, which installs libvhdi-utils alone (src/.devcontainer/Dockerfile:39).

The 4K gap cannot be closed by an integration fixture at all, and phase 16 therefore closes it at crate level. chunk_ratio is 2^23 * logical_sector_size / block_size, so reaching a second chunk group at 4096-byte sectors needs 65536 payload blocks -- a 64 GiB virtual disk with a 512 KiB BAT, where a crate fixture declares that size and materialises only the blocks it lists. And nothing authors a 4K-logical-sector VHDX at any size: qemu-img create -f vhdx -o help on 10.0.13 offers block_size, block_state_zero, log_size, size and subformat, and instar create's vhdx -o parser takes block_size and nothing sector-shaped. One 4K case does now exist, added by phase 15's second review round, and it is single-group with a single block.

The oracle was measured against the project's own goldens before phase 16 was planned, because everything in the phase rests on it. Driving set_parent() over the three real chains in instar-testdata, libvhdi agrees with vhd-diff-aligned-composed and vhdx-diff-composed over all 16 MiB, and disagrees with vhd-diff-mixed-composed in exactly one sector -- sector 2, where it returns zeros and the golden holds the parent's pattern. That is the unmasked-shift defect described above, and its shape is worth recording: the symptom is zeros, not the child's data, because the child never wrote the sector it is being read from. * Phase 17, documentation. Separate because phase 10 will have documented a refusal that phase 14 removes: at minimum docs/chain-discovery.md, docs/chain-config.md and the read-side rows and divergence notes of docs/format-coverage.md. Both chain pages already exist (209 and 210 lines, describing qcow2 chain discovery), so phase 17 extends them rather than authoring them; phase 10's survey confirmed this on 2026-09-26. Phase 11 takes two pieces of this early, because they stop being true the moment it lands rather than when phase 14 does: docs/chain-discovery.md's backing-file-support table and its Known limitations section, one of whose two stated reasons was already falsified by vhdx::posix_relative_path before phase 11 began, and docs/info.md's claim that --chain stops at one image. The compose-side rows stay with phase 17.

Corrected by phase 17's survey on 2026-10-13. The bullet above points at work phase 14 did. Commit b0ba64b8 ("Say what composes and what does not.", #630) wrote the per-operation pages, the chain-discovery capability and its Known limitations invariant, the compose section of docs/format-coverage.md, and four further pages this bullet never listed; phase 15 then corrected the refusal justifications. So docs/chain-discovery.md is current and out of phase 17's scope entirely, and the chain pages are 413 and 283 lines rather than the 209 and 210 recorded above. What is genuinely left is four false statements, two of them in files this bullet does not name: ARCHITECTURE.md:131-135 still says instar "refuses to compose one on read" and links this plan for the deferral; docs/chain-config.md:267 is headed "VHD (future)" for a reader built in phases 12 to 14, and predicts an actual_size dependency the implementation does not have; docs/format-coverage.md's read-side rows show unmarked ticks for the three operations that refuse a differencing source; and docs/info.md:112-117 defers a live limitation to this plan, which has finished without fixing it.

Two specifics phases 12 and 13 must confront, both already visible in the crates:

  • VHD differencing selects between child and parent at sector granularity, not block granularity: the per-block sector bitmap says which sectors of an allocated block are the child's. VhdState today computes the bitmap's size only to skip past it (src/crates/vhd/src/lib.rs:1513-1516, stored as block_data_offset at :1522 and applied at :1615-1616) and never reads a bit, which is right for a dynamic disk and wrong for a differencing one. Line references corrected by phase 12's survey on 2026-09-30; the claim itself still holds, and the vhd crate contains no sector-bitmap reader of any kind.
  • VHDX carries the equivalent in its sector-bitmap BAT entries, which the current walker deliberately skips (src/crates/vhdx/src/lib.rs:1820), and it treats PAYLOAD_BLOCK_PARTIALLY_PRESENT as fully present data (:1408, :1438, :1487) -- a v1 simplification that is only safe while differencing images are rejected outright.

Whether map's per-extent depth field participates is a scoping call for phase 14's plan: PLAN-map.md deferred backing-chain depth composition for qcow2 as well, so VHD/VHDX depth may reasonably follow qcow2's rather than lead it.

Phase 7 is the smallest of the implementation phases: the host already attaches the backing file as input device 0 when -b is given (run_create_nonraw, src/vmm/src/main.rs:16816), so the guest can read the parent's footer for its unique id and timestamp without any new call-table primitive. That is the fact that makes this plan tractable, and phase 1 confirmed it still holds: step 1b read the same function and found that it opens the host-resolved parent but embeds typed_backing.as_bytes() verbatim in what the guest receives (:16914-16964). That single typed string is also what forces the single-locator-entry answer in open question 3.

Phase 7 also inherits two guards it must not remove blindly. Phase 5 added an explicit backing refusal to the create operation's ImageFormat::Vhd arm and phase 6 adds the matching one for ImageFormat::Vhdx, because each emitter became able to write a child whose parent identity is all zeros before the guest could read a real one. Both guards are covered by test_create_vhd_and_vhdx_reject_backing in tests/test_create.py, added during phase 6's review round: it asserts each format refuses -b with "invalid option for target format" and writes no child file. So phase 7 will get a failing test if it removes a guard too early -- which was not true when this paragraph was first written. Issue #570 (POSIX paths emitted under Windows-only platform codes and the absolute_win32_path key) must be settled before either guard comes off, since removing them is what first exposes that choice to a user. Phase 6's review raised the same point from the planner's side: neither plan_vhd nor plan_vhdx validates backing.format, so a direct crate caller can ask for a differencing VHDX naming a qcow2 parent and get a well-formed image no consumer can compose. That is not an open choice -- open question 7 above resolved it on 2026-09-05 ("yes, with a typed error otherwise"), and phase 7 wires that error. Phase 7's plan also improves on the framing: config.backing_format is only populated when the user passes -F (src/vmm/src/main.rs:16975-16977), while the guest can detect the parent's real format from its header, so the check is on the detected format rather than on the hint.

Constraints that apply throughout

  • Guest binaries stay under the 768KB per-operation cap (make check-binary-sizes). Measured during phase 7's survey, the create op uses 65,224 bytes of it -- 8.5% -- and already depends on both the vhd and vhdx crates, so this is a regression check rather than a budget question.
  • The format crates are no_std and panic-free. Every offset and length taken from an image is bounds-checked before use; the existing qcow2 and vmdk crates are the pattern.
  • Parent locator paths are untrusted. Nothing in the guest ever opens one, and the host applies the same resolution rule it applies to qcow2 backing references.
  • Every phase that changes user-visible behaviour updates the documentation that describes it in the same pull request. Phase 10 exists for the cross-cutting pages, not as a licence to leave the per-phase pages stale.

Agent guidance

The canonical guidance -- execution model, planning effort, step tables, model roster, review checklist -- is in PLAN-TEMPLATE.md, and this plan follows it rather than restating it. What is specific to this plan:

  • Execution model. All implementation work is done by sub-agents; the management session plans, reviews the actual files rather than the sub-agent's summary, and commits.
  • Planning effort. Phases 1, 3, 4, 5 and 6 are high effort: they turn on format-spec interpretation, an architectural decision about read behaviour, or emitting structures no reference implementation in reach will double-check for us. Phases 2, 8, 9 and 10 can be planned at medium effort with good briefs. Phase 7 is high effort only because it touches the guest/host boundary; the change itself is small. Phase 11 is high effort despite a small diff: it decides what the host may resolve out of an untrusted image, and it has to preserve an invariant phase 4 established deliberately -- that a refusal never depends on whether a parent file happens to exist.
  • Model choice. Skew to opus for the emitters and the parse layer. Phase 5 and 6 briefs must name the exact byte offsets and the checksum algorithm, because a plausible-looking wrong offset in a format nobody else validates is precisely the failure this plan is exposed to.
  • A standing warning from this repository's history. Agents assert plausible-but-wrong format and tool capabilities. Require cite-or-measure for every claim about what libvhdi, Hyper-V or qemu accepts, and re-measure a sample in the management session before it is written into a brief.

Administration and logistics

Success criteria

We will know this plan has been implemented because:

  • instar create -f vpc -b PARENT -F vpc child.vhd SIZE and the vhdx equivalent produce images the phase 1 oracle resolves against their parent, with content matching what instar intended.
  • No instar op silently composes a differencing image as though it had no parent, at any commit in the plan: phase 4's refusal and then phase 11's composition are applied uniformly across info, check, convert, compare, dd, bench, map and measure. Phase 4's survey corrected two assumptions here: map already refuses (commit eb6e23f, 2026-06-03) and is the precedent phase 4 generalises rather than outstanding work, as does resize; and dd is not an operation -- run_dd (src/vmm/src/main.rs:13958) calls execute_convert, so it shares convert's guest binary and inherits its behaviour.
  • instar convert -O raw on a differencing child produces the same bytes as the phase 1 oracle's composition of the same chain -- driven through the python3-libvhdi binding, since there is no vhdiexport -- for chains instar wrote and for chains it did not, and instar info --chain walks a VHD or VHDX chain the way it walks a qcow2 one.
  • crates/vhd and crates/vhdx parse parent locator structures and are clean under the new fuzz targets, including the adversarial fixtures from phase 2.
  • make instar builds, make lint is clean, make check-binary-sizes passes, make test-rust and make test-integration pass, and pre-commit run --all-files passes.
  • docs/create.md, docs/format-coverage.md (both the output side table and the divergence notes), docs/quirks.md, docs/resize.md, docs/guest-architecture.md, docs/chain-discovery.md, docs/chain-config.md, ARCHITECTURE.md and CHANGELOG.md describe what shipped, and the false "differencing with backing chains" input claim at CHANGELOG.md:2314 is reconciled by a current statement of what is actually supported. (Phase 10's survey corrected this reference: it read :1922 until 2026-09-26, which is a line about vhdx parent virtual_size recovery. The claim had drifted 392 lines.)
  • The push audit in phase 18 has run PUSH-AUDIT.md over the union of the merged ranges, and its findings are resolved or declined in writing.

Documentation index maintenance

docs/plans/index.md carries a row for this plan in the Master plans table, and docs/plans/order.yml carries an entry for the master plan only. Phase files are linked from the index row and from the Execution table above as they are written, and are not added to order.yml. When every phase is complete the index status becomes Complete.

Future work

  • resize of a differencing image, which needs the parent-locator update path this plan builds (docs/resize.md:215).
  • rebase for differencing VHD/VHDX -- repointing a child at a new parent -- which PLAN-rebase-commit.md:236 deferred for want of exactly this parse and emit layer.
  • The other two leaves of the same 2026 goal: multi-extent VMDK output, blocked on multi-output-device support in the call table, and vmdk/vpc/vhdx preallocation, blocked on a per-format BAT population pattern.
  • Giving created VHD and VHDX images a real identity (#566). plan_vhd writes an all-zero footer unique id and vhdx::build_header derives the DataWriteGuid from the sequence number, so every image instar creates shares one identity and the parent-identity check is vacuous for chains instar wrote end to end. Deferred by decision 2 of the phase 5 plan: it needs a host-side entropy source passed through the call table, which is the one ABI change this plan is built to avoid. The negative identity test therefore uses a third-party parent; phase 7 built it, so this is a note on the constraint rather than outstanding work.
  • Differencing-aware check, once a chain can be resolved: today check refuses VHDX differencing and validates a VHD differencing child as if it were dynamic.

Outstanding work this plan filed and did not close. Twenty-six issues this plan's phases filed are still open. Phase plans are not read again once a plan retires, so this list -- not the phase plans that found them -- is the record, and it is grouped by what a reader would fix together rather than by the phase that happened to find it. Enumerated against f4a4961f;

566 above is the twenty-sixth. Issue #375 is cited by a phase

plan as precedent but predates this plan by three months and is not its deferral, so it is deliberately absent.

Data correctness and safety. #653 and #654 are the sharper pair: host_offset == 0 is interpreted ad hoc per branch instead of each subcluster being classified once into the types qcow2 defines, so an extended-L2 Standard(0) serves the image's own header to the guest as payload and exits 0, and an UNALLOCATED_PLAIN subcluster zero-fills rather than descending to the backing file on well-formed images, in both I/O arms. They share a root cause and want fixing together. Alongside them, #613 (the VHD chain read arm does not cap a chunk at the block boundary), #601 (chain devices past MAX_CHAIN_DEVICES are dropped with only a debug log) and #660 (TOCTOU between backing-path validation and the backing-file open).

Parent locators and format interop. #602 and #661 are the same defect written up twice, seventeen days apart, and that is worth knowing before either is worked: phase 11 filed #602 as a host-ABI gap (the locator table is parsed in crates/vhd but nothing in InfoResult carries it), and phase 18's audit filed

661 as dead code (preferred_locator, decode_path and

raw_path_bytes have no production caller). Both reach the same interoperability consequence -- a third-party child with an empty unicode name and a valid W2ru/W2ku entry resolves under Hyper-V and not under instar -- and the duplication is itself evidence for keeping this list: #602 was recorded only in a phase plan, so by phase 18 nobody was looking at it. One should be closed as a duplicate of the other. With them: #662 (locator parse laxity -- unchecked LocatorType, missing length validation), #608 (info --chain does not check a resolved parent is the same format as its child), #583 (the VHD footer checksum is never verified, so any conectix block is accepted as a footer), #576 (the VHDX metadata parser does not detect overlapping metadata items, SPEC(VHDX) 2.6.1.2) and #658 (info reports an unresolvable "actual path" for a Windows-converted chain).

Chain parity, and refusals whose stated reason is no longer true. #641 (map refuses a chain qemu-img composes), #642 (measure answers only the top layer of one) and #643 (whether check and measure should still refuse a differencing child at all); phase 15 established by measurement against qemu-img 10.0.13 that the first two are qcow2-chain parity work which PLAN-map.md and PLAN-measure.md each deferred before reaching Complete, rather than differencing work. Then #628 (create -b refuses a differencing backing file for a reason that is no longer true) and #632 (safe-mode rebase onto a raw new backing always fails as an unparseable overlay header).

Checks that report on less than they examined. This is the shape that recurred most across the plan, in the instrument rather than the product, and phase 18 found its own audit scripts were an instance. Still open: #590 (the differential fuzzer silently skips libyal cross-checks when a tool is missing), #593 (extract-fuzz-corpus.py silently drops 21% of manifest fixtures over MAX_SEED_SIZE), #651 (nothing cross-checks the three fuzz target lists), #663 (checks documented as guards that run nowhere, or cannot fail), #664 (fuzz_chain_compose cannot reach the VHD or VHDX compose arms, so the target named for composition never composes) and #626 (an unverified claim: the non-differencing VHDX BAT entry count may over-declare by one). Anything that ships a new guard during this group's cleanup should be mutation-tested rather than read, which is what turned these up.

Structure and documentation debt. #665 (compose logic duplicated across four layers, with nothing pairing a raiser to a renderer) and #592 (plan phase, step and decision references in docs and code). #592 is the tracking issue for a convention rather than a defect, and the scale is worth stating because the issue does not: measured on f4a4961f, 41 PLAN-differencing citations remain in src/ and crates/, 105 counting every PLAN-*.md, so that part is a sweep. A case-insensitive grep for bare numbered-phase references in .rs files matches 407 lines, which needs triage instead -- among them src/vmm/src/main.rs:7034 and :7038, which are /// doc comments and so reach users as CLI help reading "Phase 3 only supports 512 because the" and "Documented; phase 5 may relax this". Not covered by any issue: the 4096-byte-sector VHDX limitation that phase 13 found and phase 16 confirmed cannot be fixtured appears in plan files only, so it will vanish when they stop being read.

Bugs fixed during this work

  • docs/create.md claimed vpc and vhdx honoured backing_file and backing_fmt, and marked both "Yes" for backing support, where both planners reject a backing reference outright. Fixed on this branch in a93615d, ahead of the phase 1 plan file.
  • instar compare reports "Content mismatch at offset 0!" for a differencing VHDX compared against itself, because VhdxState::init's refusal happens in the guest rather than on the host and compare reads the resulting absence of content as a difference. A format refused on the host, such as bochs or qed, refuses cleanly by name. Filed as #548; phase 4 fixes it alongside #547.
  • The silent parent-ignoring read of differencing VHDs is a live correctness defect, not merely a missing feature: convert produces a wrong image and exits 0. Filed by step 1d as issue #547, "VHD differencing (disk type 4): convert -O raw silently composes wrong data, exits 0", labelled bug; phase 4 fixes it. There were no open issues on this surface when the plan was written.
  • instar-testdata/custom/format-coverage/vhd-differencing.vhd is a disk_type = 4 marker with an empty parent name and eight zeroed locator entries, so it does not exercise what its name implies. Phase 2 replaces or supplements it and records what the old fixture was actually testing.

Back brief

Before executing any step of this plan, please back brief the operator as to your understanding of the plan and how the work you intend to do aligns with that plan.

📝 Report an issue with this page