Streaming image input over HTTP and HTTPS¶
Status: Not started¶
Prompt¶
Before responding to questions or discussion points in this document, explore the instar codebase thoroughly. Read relevant source files, understand existing patterns (VMM structure, guest operation layout, shared crate conventions, call table ABI, format parsing, test infrastructure), and ground your answers in what the code actually does today. Do not speculate about the codebase when you could read it instead. Where a question touches on external concepts (QCOW2, VMDK, VHD/VHDX, LUKS, KVM, virtio, disk image formats), research as needed to give a confident answer. Flag any uncertainty explicitly rather than guessing.
Consult ARCHITECTURE.md for the overall system structure
(host VMM, KVM guest, call table, device emulation).
Consult AGENTS.md for build commands, project conventions,
code organisation, and the security model summary. Consult
docs/ for format-specific documentation (docs/qcow2/,
docs/raw/, etc.) and docs/commentary/ for architectural
decisions and design rationale.
Plan file conventions (shared block; do not edit -- the canonical
copy lives in shakenfist/development at
templates/shared-blocks/plan-file-conventions.md):
- All planning documents live in
docs/plans/. - Detailed planning gets one plan file per phase. Phase files are
named for their master plan, sit in the same directory as it,
and append
-phase-NN-descriptivebefore the.mdextension. - The master plan tracks its phases in a table under its Execution
section.
Mergedis last, and the push audit is the last row, for the reasons given inplan-push-audit-phase:
| Phase | Plan | Status | Merged |
|---|---|---|---|
| 1. Schema migration | PLAN-thing-phase-01-schema.md | Not started | |
| 2. Public API | PLAN-thing-phase-02-api.md | Not started | |
| 3. Push audit | - | Not started |
- One commit per logical change, and at minimum one commit per phase. Unrelated changes are not batched into a single commit. Each commit is self-contained: it builds, passes tests, and has a message explaining what changed and why.
Situation¶
instar takes every input as a local path. The host opens the file, exposes it to the guest as a virtio-block device, and never looks inside it; the guest parses the format. Nothing in that design assumes the bytes come from a filesystem, and the code that would have to change is small and already in one place.
The I/O boundary. src/vmm/src/backing.rs holds BackingStore:
a File, a capacity, and read_at(offset, &mut [u8]) / write_at
/ sync / size. Every device is built on it -- 36 construction
sites and 26 VirtioBlockDevice::new calls -- and every one of them
already addresses bytes by offset. A second backing kind with the
same methods is invisible above open().
No network, and nothing to build a partition from. The VMM's
dependencies are kvm-ioctls, kvm-bindings, vm-memory, virtio-queue,
virtio-bindings, vmm-sys-util, clap, libc, log, env_logger,
guest-protocol, shared, the format crates, toml, dirs, serde and
rand. There is no HTTP client, no TLS stack and no URL parser in the
process that holds /dev/kvm. There is also no privilege-separation
machinery: the VMM never forks a helper (version.rs shells out to
qemu-img --version, and that is all), installs no seccomp filter,
applies no Landlock ruleset, and sets no rlimits.
What the guest actually reads. Measured with strace on the
prebuilt binary running convert -O raw, counting reads on the
input descriptor:
| Source | Input reads | Pattern |
|---|---|---|
8 MiB qcow2 written by qemu-img convert |
131 x 64 KiB | offsets strictly ascending |
| 8 MiB qcow2, ten clusters written out of order | 13 x 64 KiB | 0 196608 262144 393216 655360 917504 524288 786432 589824 851968 458752 720896 327680 |
Three things follow. Requests are 64 KiB, because convert's
--sector-size defaults to 65536. Unallocated regions are never
read, so a sparse source costs only its allocated clusters. And the
order is whatever the image's own metadata dictates, including
backward jumps, which the host cannot predict because the host does
not parse the format -- that is the security model, not an
oversight. Some paths still issue 512-byte requests (issue #860 for
create), so a remote backing store has to coalesce rather than
assume 64 KiB.
The guest does not block on I/O; it spins.
src/core/src/virtio.rs do_request submits one request, polls the
used ring for MAX_WAIT_ITERATIONS = 100_000_000 iterations of
spin_loop(), and calls send_error("timeout", ...) if the count
runs out. Measured on this host, 100M pause instructions take
1.3 s. The budget is counted in iterations, not time, so a
descheduled vCPU does not consume it; what consumes it is the host
taking longer than that to answer one read. Queue depth is one.
Unix-socket limits, measured here. net.core.wmem_max is
212992, so a SOCK_SEQPACKET socketpair's SO_SNDBUF saturates at
425984 however much is requested: a single 256 KiB datagram sends,
and a single 1 MiB datagram fails. Any design that carries a block
of image data in one datagram is capped near 256 KiB on default
sysctls, and that limit is a sysctl the user can lower.
What the host already reads for itself. Backing-chain discovery
(src/vmm/src/chain.rs) runs the sandboxed info operation once per
chain level, and checks each discovered path against an allowlist
(get_backing_allowlist, default $IMAGE_DIR) before touching the
filesystem. Mutating operations probe the target on the host with
plain std::fs before launch: probe_resize_target,
probe_amend_target, probe_bitmap_target,
probe_rebase_target. Configuration layers /etc/instar/config
then ~/.config/instar/config, and the allowlist is settable in
both, so the user-writable file can widen the system one.
Confinement available, measured on this host. Landlock is
compiled in and present in the active LSM list, unprivileged user
namespaces are enabled (max_user_namespaces 114274), and
/dev/fuse, fusermount3, bwrap and curl all exist. Kernel
6.12 carries Landlock ABI v4 or later, which restricts connect by
TCP port as well as filesystem access. None of this is portable
across the distro matrix, so what the running kernel refuses has to
be reported rather than assumed.
Mission and problem statement¶
Let read-only operations take an https:// input, start work before
the transfer finishes, and fetch only the bytes the image's own
metadata asks for -- without putting a line of network, TLS or URL
code in the process that holds /dev/kvm.
Why this is not just a download. Spooling first already works
and needs no code: curl -o image.qcow2 ... && instar convert ....
This feature earns its place only through the two things spooling
cannot do -- starting work immediately, and never transferring what
is not read. The fragmented measurement above would move 900 KiB
instead of 8 MiB.
Why the buffer is the easy part. A bounded block cache over range requests is about a day's work. Three other things are not.
- The guest's wait is a fixed spin budget. A cache miss that takes longer than about 1.3 s -- a TLS handshake, a cold origin, a 429 -- is reported by the guest as a hard I/O failure rather than retried. Nothing downstream can be made reliable until the guest waits properly.
- Latency is serialised. Queue depth is one and the guest blocks per request, so 10 GiB at 64 KiB is about 164,000 sequential round trips: thirteen minutes of pure waiting at 5 ms each. The host has to fetch ahead of the guest, which means the fetch must be asynchronous and the cache must usually hit.
- Posture. An HTTP client in the VMM puts a response parser, a
redirect follower, a DNS resolver and a TLS stack inside the
process that holds
/dev/kvm-- and holds the invoking user's entire ambient authority, which running as a non-root user does not reduce. Half the objection is supply chain rather than memory safety:reqwestpulls tokio and several hundred crates into a binary whose dependency list is fifteen lines long and whose.deband.rpmdependencies are generated from it.
The argument that makes a cheap partition sufficient. instar already assumes its input bytes are hostile; that is what the sandbox is for. A fetcher confined so that it can do nothing but return bytes therefore degrades, when compromised, to the attacker controls the image -- the baseline assumption, already handled. The fetcher does not need to be trusted, so it does not need to be a guest. It needs to be unable to do anything else.
Shape. A separate instar-fetch process holds every line of
network code. It is started before the VMM opens /dev/kvm, so it
never inherits that descriptor; it is given one socket and no other
capability; and it is confined with Landlock, seccomp, namespaces
and rlimits as far as the running kernel allows, saying so when the
kernel allows less. The VMM speaks a fixed-layout,
offset-addressed byte protocol to it and parses nothing else.
Out of scope. Remote output; any in-place mutation over the network; authentication; remote backing-file references; and protocols other than HTTP and HTTPS. Each is refused by name rather than left to fail obscurely.
Open questions¶
All nine are resolved. Question 5 was settled by measurement on 2026-11-18; the operator settled the other eight the same day, taking the planner's recommendation in every case. Each question keeps the reasoning that was offered for it, which is now the rationale of record rather than an argument for a decision still to be made -- a reviewer asking "why a framed protocol and not a virtqueue?" should find the answer here, along with what was rejected and on what grounds. Nothing in the phase table is waiting on a decision; phase 2 can be planned as written.
A decision recorded here is not immovable. Questions 2 and 4 are
the two most likely to be revisited by measurement -- the datagram
ceiling and block size against phase 6's round-trip counts, and the
HTTP client if ureq turns out to want something the fetcher cannot
give it -- and both say so in their own text.
- A virtio-derived protocol over the socket, or a plain framed one? Resolved 2026-11-18: a plain framed protocol that borrows virtio-blk's model and not its mechanism.
What virtio-blk is for does not survive the move. A descriptor
is a guest-physical address and a length; the ring lives in
memory both sides map; the doorbell and the used ring exist so
that neither side has to copy. Over a Unix socket none of that
exists. Transliterating the structures keeps the awkward half --
sector addressing, a status byte in its own descriptor, F_NEXT
chains -- and loses the entire point.
Shared memory could be reintroduced, and should not be.
SCM_RIGHTS passes a memfd and an eventfd over the socket;
ioevent.rs already does eventfd notification and virtio-queue
already implements rings over vm-memory, so a real virtqueue
between the VMM and the fetcher would work. It is the version
that deserves the name, and it is still the wrong answer: it
hands the untrusted fetcher a writable mapping shared with the
privileged process, and whoever controls the ring controls
indices the VMM dereferences. That is precisely the
memory-corruption channel the partition exists to remove.
Defending it means writing adversarial-ring validation in
privileged code -- the same class of work as the existing virtio
boundary audit -- to save a copy that is noise: 256 KiB of
memcpy is tens of microseconds against tens of milliseconds of
network.
The asymmetry that decides it. The payload is bytes already
assumed hostile, so copying them costs only cycles. Control
structures are not hostile-tolerant. Pay the copy, give the
fetcher one socket and no shared memory, and docs/security.md
can state that the partition is total.
What is worth borrowing. The model: capacity discovered at init, offset-addressed reads, a typed status rather than a bare errno, an explicit flush -- and, from the used ring, out-of-order completion. A request id with several requests outstanding is what lets the fetcher prefetch without shared memory, and it is the one piece of virtqueue thinking the stream protocol genuinely needs.
This shape is NBD, and we should still not speak NBD. NBD is
offset-addressed reads over a socket with request handles and
out-of-order replies, and nbdkit's curl plugin is an
off-the-shelf implementation of nearly this whole feature.
Speaking real NBD means parsing a negotiated protocol with
option haggling and structured replies from an untrusted peer in
the privileged process, which is a far larger surface than the
few fixed-size headers here. Steal the shape; do not link the
protocol. Phase 2's plan should record the comparison, because a
reviewer will ask.
And not protobuf, unlike guest-protocol. That crate uses
micropb because the guest's serial channel carries
variable-shape result messages. This channel carries fixed-size
headers and raw bytes, so it should be #[repr(C)] headers with
explicit bounds checks and no serialisation library at all. It is
the only new untrusted parse in the privileged process and it
should be small enough to fuzz exhaustively.
-
SOCK_SEQPACKETorSOCK_STREAM, and what block size? Resolved 2026-11-18:SOCK_SEQPACKET, 64 KiB default, 256 KiB ceiling negotiated at init. SEQPACKET makes message boundaries the kernel's problem, which removes framing desynchronisation from the privileged parser entirely. 64 KiB matches the guest's own request size, and readahead comes from many outstanding requests rather than larger ones. The measuredwmem_maxof 212992 caps a single datagram near 256 KiB here and will be lower on some machines, so the ceiling is negotiated rather than assumed. If phase 6's measurement shows round-trip count dominating, phase 6 revisits it --SOCK_STREAMwith fixed headers, or scatter-gather across several datagrams -- rather than phase 2 guessing now. -
Where do the cache and the readahead live? Resolved 2026-11-18: in the fetcher. The VMM's
read_atstays one round trip, and the fetcher owns the block cache, the coalescing of 512-byte requests, and the asynchronous readahead, issuing overlapping HTTP requests without the VMM knowing. A compromised fetcher serving stale or wrong cached bytes is still only serving wrong bytes. Putting the cache in the VMM would mean the privileged process holding and indexing attacker-influenced data and scheduling the prefetch, which is more privileged code for no gain. Consequence: the VMM's memory does not grow, and any--cache-sizecontrol is a fetcher argument. -
What implements HTTP inside the fetcher? Resolved 2026-11-18:
ureqwithrustls, in the fetcher's own crate, never a dependency of the VMM binary. It is blocking, which suits a thread-per-outstanding-request fetcher; it does not pull tokio; and rustls is memory-safe Rust, so the residual risk is low even before confinement. Being pure Rust it also adds no shared library, which matters becausecargo deb's$autoandgenerate-rpm'sauto-reqscan the packaged binaries: a C TLS stack would show up in the package metadata, and rustls will not. Phase 5's plan should record the rejected alternatives:reqwest(tokio and hundreds of crates, which still have to be audited and packaged even though the VMM does not link them);libcurl(C in the fetcher, but mature and distro-patched -- the reasonable second choice); and exec'ingcurl --rangeper block (zero HTTP code anywhere in instar and the most auditable of all, but one process per cache fill, which the operator has ruled out as the primary design). -
Does the spin budget already cause the kasm
test_convertflake? Resolved 2026-11-18 by measurement: the mechanism is real, it explains one of the cluster's two messages, and the other was already fixed. Injecting a single delayedreadwithstrace -e inject=read:delay_enter=...into a release build ofdevelopmakes the guest declareerror op=timeout device=... status=57005once the host takes longer than somewhere between 0.5 s and 1.0 s to answer one request, against the 1.3 s the budget measures on an idle host. It fires on output writes as well as input reads, because both go through the samedo_request.convertthen reportsError: "convert operation failed"with exit 1, says nothing about I/O unless--verboseis passed, and leaves a truncated output file behind -- 524288 bytes of an 8388608-byte conversion. That is the cluster's first message.comparehandles the same timeout correctly: "compare: reading the second image, ref.raw, failed at virtual offset 327680 (0x50000); the images were not compared", exit 4, which is qemu-img's read-error code. So the cluster's other message, "Content mismatch at offset 0!", is not this. It was issue #807, fixed on 2026-11-12 by 360bfdf4: a pre-fix binary reproduces it under the same injection and a current one does not, which also means flake observations recorded before that date should not be assumed still live. Phase 1 therefore starts from a reproduction rather than a hypothesis, and carries two defects worth fixing with no network in sight: the budget itself (#923), and convert's silence about an I/O failure (#924). -
A server with no
Rangesupport: refuse, or spool? Resolved 2026-11-18: refuse by name, with--spoolas an explicit opt-in that says what it is doing and where it is writing. Silently spooling turns a streaming operation into a full download plus a disk write, which is what the user would have chosencurlfor. Note the asymmetry:Accept-Rangesis advisory, so the only reliable probe is a ranged request checked for206, which phase 5 does once at open. -
Credentials. Resolved 2026-11-18: none in v1. A URL containing userinfo is refused; no
.netrc, no bearer token, no--header. The fetcher then never holds a secret, which is one fewer thing a compromise can take. Signed URLs carry their credential in the query string and work anyway, covering most object-store cases -- but they must then be kept out of verbose output and out of error strings, which phase 8 owns. -
How is the fetcher found and packaged? Resolved 2026-11-18: the same pattern as the guest
.binfiles -- probe the executable's own directory first for developer builds, then/usr/lib/instar/. It is a workspace binary built bymake instarand added to the.deband.rpmasset lists at mode 755. -
Remote output, and remote backing files. Resolved 2026-11-18: both out of scope, and both refused by name. Output stays local because every writer seeks backwards. Backing files stay local as policy rather than as a limitation: an image-supplied URL is attacker-controlled data, and following one is server-side request forgery from a process the attacker is trying to reach. The only URL instar ever fetches is the one on the command line. Phase 7 enforces it; phase 10 writes it down.
Execution¶
Push audit phase (shared block; do not edit -- the canonical
copy lives in shakenfist/development at
templates/shared-blocks/plan-push-audit-phase.md):
- Every master plan ends with a phase that runs the repository's
PUSH-AUDIT.mdover the whole plan's work. It is the last row of the Execution table and it is not optional. The rule binds every plan that carries the phase, which is decidable from the plan file alone: a plan that is alreadyComplete,AbandonedorSupersededand does not carry the phase is not reopened to acquire one, and a plan that has the phase runs it even if it reachesCompletebefore the phase does. - That phase audits the accumulated diff of every phase in the plan against the default branch, not the diff of the last phase alone. Auditing one phase at a time would miss what the phases did to each other -- the duplicated helper that only exists once phases three and six have both landed, the doc page that phase two made wrong and phase five never revisited.
- Once the plan's phases have merged, a diff against the default
branch is empty and would read as a clean audit. The range is not
reliably derivable after the fact either: unrelated work lands on
the default branch between phases, so anything anchored on "since
the plan file appeared" is far too wide. It has to be recorded. As
each phase lands, what put it on the default branch goes into the
plan: the merge commit of its pull request, whose diff against its
first parent is the whole of what landed, or -- where the phase
landed directly -- every commit of the phase, or its
first..lastrange. A single commit is only ever enough when it is a merge commit. - Where the Execution phases are a table, that record is a
Mergedcolumn, added last so that a row which omits it still reachesStatus; where they are prose sections it is aMerged:line in the phase's own section. TheStatuscolumn keeps its single vocabulary term and nothing else (seeplan-status-vocabulary). A phase that landed in another repository records<repo> <sha> (#pr)and is audited against that repository's default branch, as part of the pull request that lands it; the plan's own push-audit phase cites that audit rather than re-running it. - Phases that landed before the plan started recording them are
reconstructed rather than left blank. Recover what you can from
gh pr list --state mergedandgit rev-list --first-parent, and say in the plan that the range was reconstructed. Do not trust a path-filteredgit logon its own: it lists the commits that touched a path without saying which arrived directly and which arrived inside a pull request, and recording a commit that came in under a merge audits one commit of that pull request rather than the pull request. A reconstructed record may be a summary table in the audit phase's own section rather than a column or a line in the Execution table, which keeps retrospective archaeology out of a table that tracks live status. Where a phase accreted over months of unrelated commits and no range is recoverable, say that instead and name the paths the audit read -- an audit that says what it could not scope is a result; one that silently audits nothing is not. - Findings land as their own pull request against the default branch, and the plan is not complete until they are resolved or explicitly declined in writing. A finding that is declined says why, in the plan, where the next reader will find it.
- Where the audit finds nothing, record that in the plan in one sentence. It is a real result, and a run of them is the evidence for making the phase conditional rather than mandatory.
- A repository with no
PUSH-AUDIT.mdstill carries the phase, and the phase says that the runbook does not exist yet and what was done instead. Silently omitting it is what let the audit go untriggered for as long as it did. What is done instead is at least the checklist below, applied to the recorded ranges: it is a floor, not a substitute for a runbook, and a plan adds the questions its own repository's blast radius raises. Each item gets a verdict, or a sentence saying why it does not apply. Do not borrow another repository's runbook instead: a runbook is written for one repository's blast radius. - Every recorded landing takes one of the forms above: a merge
commit with two parents, whose diff is read against its first
parent; every commit of a direct landing, or its
first..lastrange; a landing in another repository, whose own audit is cited; or a statement that no range is recoverable, naming the paths the audit read. - Anything the diff added that runs unattended -- cron, a timer, a scheduled workflow, a self-update -- reports its own failure somewhere a human sees, or is named as not doing so.
- No secret or token entered any repository, and no private hostname or IP address entered a repository more public than the one it came from.
- Anything the diff removed or renamed has no consumer left, checked by grep across every repository that could consume it rather than by assumption, with the greps stated.
- Where the repository publishes something -- an image, a release, a package -- a broken change fails closed: one that publishes nothing is recoverable, one that publishes something wrong is not.
The phase order is chosen so that the two hard problems are solved
before the easy one is attempted. Phase 1 is first because nothing
downstream is reliable while the guest reports a slow read as a
failure, and because it is worth landing on its own merits. Phases 2
to 4 then build and confine the whole process boundary with no
network code anywhere in the tree, so that the protocol and the
sandbox are exercised where a failure cannot be blamed on a server;
they also leave a file:// backend behind, which is the oracle
every later phase tests against. HTTP arrives only at phase 5, once
there is a confined process to put it in.
| Phase | Plan | Status | Merged |
|---|---|---|---|
| 1. Guest I/O wait | PLAN-http-input-phase-01-guest-wait.md | Not started | |
| 2. Fetch protocol and a file-backed fetcher | PLAN-http-input-phase-02-protocol.md | Not started | |
| 3. Remote backing store | PLAN-http-input-phase-03-backing.md | Not started | |
| 4. Confinement | PLAN-http-input-phase-04-confinement.md | Not started | |
| 5. HTTP backend | PLAN-http-input-phase-05-http.md | Not started | |
| 6. Cache and prefetch | PLAN-http-input-phase-06-prefetch.md | Not started | |
| 7. Operation scope and chain policy | PLAN-http-input-phase-07-scope.md | Not started | |
| 8. Failure semantics and UX | PLAN-http-input-phase-08-failures.md | Not started | |
| 9. Tests and fuzzing | PLAN-http-input-phase-09-tests.md | Not started | |
| 10. Documentation and threat model | PLAN-http-input-phase-10-docs.md | Not started | |
| 11. Push audit | - | Not started |
1. Guest I/O wait. Replace the iteration-counted spin in
src/core/src/virtio.rs do_request with a wait that does not
expire because the host was slow. Open question 5 settled this as a
bug fix rather than enabling work: a single host I/O slower than
about a second already fails a local operation, with no network in
sight, and the reproduction is in that question. The phase also
carries convert reporting such a failure as a bare "convert
operation failed" with no mention of I/O outside --verbose.
Touches every operation's I/O path, so it lands alone.
This section said, until the phase was planned, that the deadline
should move "to the host where it can be generous and per-request".
The phase's survey found there is no host deadline to move it to --
nothing in the VMM times a request, and the 100 ms epoll_wait
timeout in src/vmm/src/io_thread.rs exists only so the loop can
re-check its running flag -- and that neither thread which
services a virtio queue could enforce one, because both block
inside the synchronous read_at. The operator therefore decided on
2026-11-18 that the guest waits indefinitely and the host enforces
no deadline at all; the phase plan argues it, and
src/core/src/serial.rs read_config is the precedent, waiting
unbounded on the host already. The truncated output file, which this
section also used to claim, is #849 and is out of the phase's scope.
2. Fetch protocol and a file-backed fetcher. Define the wire
protocol -- fixed-layout headers, offset-addressed reads,
out-of-order completion by request id -- and build instar-fetch
serving a local file over it, with no network code at all. Records
the NBD and virtqueue comparisons from open question 1.
3. Remote backing store. BackingStore gains a second kind,
the read-only operations accept a URL, and the fetcher's lifecycle
is owned: started before /dev/kvm is opened, reaped on every exit
path. End state: instar info file:///path works end to end
through the socket.
4. Confinement. Landlock, seccomp-bpf, user and mount
namespaces, rlimits (RLIMIT_FSIZE 0 and RLIMIT_NPROC 0 are the
cheap wins), PR_SET_NO_NEW_PRIVS, PR_SET_DUMPABLE 0, and
descriptor hygiene, applied in the child before exec. What the
running kernel refuses is reported, not silently skipped. Still no
network, so the sandbox is tested against a fetcher that cannot be
blamed for network behaviour.
5. HTTP backend. Ranged GETs, sizing from Content-Length, the
206 probe, ETag/Last-Modified pinning so a source that changes
mid-transfer fails cleanly, retry with backoff, redirects refused,
scheme and host policy including a default deny for link-local and
RFC1918, and resolve-then-connect-to-the-pinned-address so a DNS
rebind cannot walk past the policy check. Verifies that
cargo tree -p instar is unchanged.
6. Cache and prefetch. The block cache, coalescing of sub-block requests, asynchronous readahead, several requests outstanding, and the measurement that says whether phase 2's block size was right.
7. Operation scope and chain policy. URLs accepted only by read-only operations and refused by name everywhere else, including outputs and image-supplied backing references. Chain discovery's per-level VM launches reuse one fetcher rather than refetching headers per launch.
8. Failure semantics and UX. Progress reporting, deadlines and
caps, the --spool fallback, resumption, and error messages that
report a failure without echoing a signed URL.
9. Tests and fuzzing. An integration HTTP fixture; a
fault-injecting server covering no-range, truncation, an ETag
change mid-transfer, 429 and a stall; a confinement test that reads
/proc/<pid>/fd and asserts the sandbox denies what it should; and
a cargo-fuzz target for the framing parser, which is the only new
untrusted parse in the privileged process.
10. Documentation and threat model. docs/remote-input.md, the
partition section of docs/security.md including what it does not
cover, ARCHITECTURE.md's component inventory -- a second process
is a change in the shape of the system -- CHANGELOG.md, and the
README's documentation links.
In this project
The Execution phases are Phase | Plan | Status table each
master plan carries, so the record goes in a Merged column
added after Status — last, so that a row which omits it still
reaches Status. Most phases here land as their own pull
request and the cell holds that one merge commit; a phase that
landed directly on develop records its first..last range
instead. The audit phase itself runs PUSH-AUDIT.md over the
union of those ranges rather than over git diff develop...HEAD,
which is empty once the phases have merged.
The obligation starts with plans written from here on, and with
any plan still open that carries an audit phase. Plans that were
already Complete when the rule arrived are not being backfilled,
so a Phase | Plan | Status table with no Merged column is a
plan that predates this, not a plan that has drifted from it.
Phase landing (shared block; do not edit -- the canonical copy
lives in shakenfist/development at
templates/shared-blocks/plan-phase-landing.md):
A plan's status and a repository's review state both live in files that every branch would otherwise rewrite. Left alone, that turns each of them into a merge-conflict hot spot, and it spends a pull request and a full CI run on a change that is entirely prose. Three rules keep them out of the way.
-
A phase is closed out in the first commit of the next phase, not in a pull request of its own. By the time the next phase branches, the previous one has merged, so its merge commit is known and its
Mergedcell can record the thing the push-audit phase actually needs. This is the only ordering that works: a phase cannot record its own merge commit, and a separate close-out pull request buys that record at the price of a round trip. The close-out sets the finished phase'sStatusandMergedcells and the plan's row indocs/plans/index.md, and it is committed before the next phase's own work, so that the branch never claims the plan is further along than the default branch is. -
The last phase closes itself out. The push-audit phase is the last row of every plan, so no next phase will carry its close-out. Where the audit raises findings, the plan is not complete until they are resolved or declined, and those land as their own pull request after the audit phase has merged -- so that pull request is the carrier, and it can record the audit phase's merge commit, which by then is known. Where the audit finds nothing there is no carrier, and no follow-up pull request is opened for the sake of one cell: the phase sets its own
Status, and the plan's index row, toCompletein its own pull request, and records noMergedcell. It is the only row permitted to omit one. The column exists so that the push-audit phase can reconstruct what to audit; the audit phase is last, so nothing ever reads its own row. -
REVIEWS.mdis not pruned or regenerated in a pull request that changes code or documentation. Editing a reviewed file stales its mark, and adding or removing an in-scope file moves the header count, but neither is the landing pull request's business.pruneregenerates the file whether or not it dropped anything, so theprune-reviewsworkflow heals both on the next push to the default branch. Pruning from a branch is also wrong more often than it is right, though not for the reason it first appears:prunecompares each stamp againstHEAD, which on a branch is the branch tip, so it drops the marks for the files the pull request itself touched while keeping marks the default branch has already pruned. Committing that state merges a review file computed from a stale tree, and can resurrect marksprune-reviewshas already removed. Accumulated staleness is reported by thereview-coverageaudit, which recomputes coverage againstHEADand raises an issue once the backlog is worth a review session.
A review session is the exception, and it is not optional
tidiness: stamp regenerates REVIEWS.md as well as writing the
marks, and the rows, the sidecars and the marks are committed
together (see docs/code-review-tracking.md). Where a repository
requires a pull request to reach its default branch, that is how
a review session lands, so "not in a pull request" is about the
kind of change, not the mechanism.
These rules assume phases land one after another. Where two phase branches are open at once, each closes out only the phase it directly follows.
In this project
instar does not track per-file human review: there is no
REVIEWS.md, no prune-reviews workflow and no
docs/code-review-tracking.md, so the third rule has nothing to
act on here until that tracking arrives. The first two rules
apply as written to the Phase | Plan | Status | Merged table
each master plan carries: the next phase's first commit sets the
finished phase's Status and Merged cells and its row in
docs/plans/index.md, and the push-audit phase closes itself out.
Plan status vocabulary (shared block; do not edit -- the canonical
copy lives in shakenfist/development at
templates/shared-blocks/plan-status-vocabulary.md):
A status cell -- in the master plan's own Execution phase table, and
in the row docs/plans/index.md carries for the plan -- holds
exactly one of these terms and nothing else:
Proposed-- written down as a concept, not yet scheduled.Not started-- scheduled, but no work has begun.In progress-- work has begun and has not finished.Blocked-- cannot proceed until something outside the plan changes. Say what, in the plan.Complete-- the work is done.Abandoned-- deliberately dropped without being done.Superseded-- replaced by another plan, which the plan names.
The term is the whole cell. No dates, no phase arithmetic, no parenthetical qualifiers, no summary of what happened: a status is read to decide whether a plan still wants attention, and prose in that column has repeatedly grown until it could no longer be read either by a person scanning the table or by tooling. Detail belongs in the plan file, and a one-line summary belongs in the index's own Intent column.
Matching is case-insensitive, so In Progress is accepted, but the
spelling above is the one to write.
Agent guidance¶
Execution model¶
Sub-agent execution model (shared block; do not edit -- the
canonical copy lives in shakenfist/development at
templates/shared-blocks/subagent-execution-model.md):
All implementation work is done by sub-agents, never in the management session. The management session is reserved for planning, review, and decision-making. This keeps the management context lean and avoids drowning it in implementation diffs.
The workflow is:
- Plan at high effort in the management session.
- Spawn a sub-agent for each implementation step with the brief from the plan, at the recommended effort level and model.
- Review the sub-agent's output in the management session. Check the actual files -- the sub-agent's summary describes what it intended, not necessarily what it did.
- Fix or retry if the output is wrong. Diagnose whether the brief was insufficient (improve it) or the model was too light (upgrade it), then re-run.
- Commit once the management session is satisfied.
This applies to all steps, including high-effort ones. If a sub-agent cannot succeed even with a detailed brief and the right model, that is a signal the brief needs improving, not that the management session should do the implementation itself.
Use isolation: "worktree" for sub-agents when the change is
risky or experimental; the worktree is discarded if the output is
unsatisfactory. For safe, well-understood changes, sub-agents can
work directly in the main tree.
Planning effort¶
Planning effort (shared block; do not edit -- the canonical copy
lives in shakenfist/development at
templates/shared-blocks/plan-planning-effort.md):
The master plan itself is always created at high effort -- it requires broad codebase understanding, cross-referencing several source files, and judgment calls about scope and sequencing.
Each phase plan states the recommended effort level for planning that phase. Phases that turn on design decisions, cross-component coordination, protocol changes, or subtle correctness questions should be planned at high effort. Phases that are mechanical, or that follow a pattern already established elsewhere in the codebase, can be planned at medium effort.
In this project
Phases involving deep protocol research, format-spec interpretation, or architectural decisions (call-table changes, new operations, new shared crates, security boundary changes) should be planned at high effort. Phases that are mechanical or follow well-established patterns can be planned at medium effort.
Step-level guidance¶
Sub-agent step guidance (shared block; do not edit -- the
canonical copy lives in shakenfist/development at
templates/shared-blocks/subagent-step-guidance.md):
Each phase plan includes a table like this:
| Step | Effort | Model | Isolation | Brief for sub-agent |
|---|---|---|---|---|
| 1a | medium | sonnet | none | One-sentence summary of what to do and which files to touch |
| 1b | high | opus | worktree | Why this needs high effort: requires understanding X to do Y |
Effort levels, from cheapest to most thorough:
- low -- Purely mechanical changes: rename, reformat, add a log line, regenerate generated code. The brief is a complete instruction.
- medium -- The plan provides enough context to follow a clear brief. The sub-agent may read a few files, but the approach is already decided.
- high -- Requires reading several files, making judgment calls, or understanding non-obvious invariants. The sub-agent needs to think about edge cases.
- xhigh -- The setting for hard coding and agentic steps: long-horizon changes, or steps where the sub-agent must both research and implement.
- max -- Correctness matters more than cost. Expect diminishing returns and occasional overthinking; reserve it for steps where a wrong answer would be expensive to detect.
Brief for sub-agent: this is the key field. Write it as if briefing a colleague who has never seen the codebase. Include what to change, which files to touch, what patterns to follow, and any non-obvious constraints.
A good brief front-loads the research the planner already did, so the implementing agent does not repeat it. Instead of "add storage functions for the new object", name the functions to add, the file they belong in, the existing equivalent to mirror (with line numbers), and any registration the change also needs.
The better the brief, the lower the effort level needed and the lighter the model that can succeed.
In this project
The external references that push a step to high effort are
format specs, the qemu-img source, and the KVM/virtio docs.
The non-obvious constraints a brief should always state are
memory layout, the 768KB guest binary cap, the no_std
requirement of the format crates, and the call table
boundary.
Format-conversion work is the usual reason to need a large-context model: it tends to span the source format parser, the destination format writer, the call table, and the host-side glue at the same time.
A worked brief for this codebase: instead of "add tests for
the QCOW2 L2 parser", write "add tests for
parse_l2_entry() in src/crates/qcow2/src/lib.rs. Use the
adversarial fixtures in
instar-testdata/adversarial/qcow2/ (cluster boundary
edges, OFLAG_COMPRESSED set with extended L2 cluster,
refcount underflow). The function takes (entry: u64,
cluster_bits: u32) and returns Option<L2Entry>."
Model choice¶
Sub-agent model roster (shared block; do not edit -- the canonical
copy lives in shakenfist/development at
templates/shared-blocks/subagent-model-roster.md):
The planner recommends which model is best suited to each step. This is a judgment call, not a rigid rule -- the right model depends on what the step requires, not on whether it is "planning" or "implementation". The models available to sub-agents are:
- fable -- The most capable model available, for the hardest reasoning and the longest-horizon work: multi-step changes a single sub-agent must carry end to end, or steps whose correctness depends on holding a whole subsystem in mind at once. It costs materially more than opus, so reserve it for steps that have already defeated opus or are expected to.
- opus -- The default for steps needing deep reasoning, architectural understanding, subtle correctness judgment (locking, state machines, migrations), or intricate implementation that would be costly to debug if it were wrong.
- sonnet -- A good default for well-briefed implementation work. Faster and cheaper than opus, and effective when the plan front-loads the research and the brief leaves no broad judgment calls to make.
- haiku -- Suitable for purely mechanical tasks: search-and-replace, regenerating generated code, adding log lines, running commands. The brief must be a near-complete instruction.
Model choice interacts with effort level and brief quality. A detailed brief compensates for a lighter model -- sonnet at medium effort with a thorough brief often matches opus at medium effort with a vague brief. The planner's job is to write briefs good enough that the recommended model can succeed.
The model also determines the context window: fable, opus and sonnet have 1M tokens, haiku has 200K. A step that must hold many files in context at once may need one of the larger-context models for that reason alone, even when the reasoning itself is straightforward.
When in doubt, skew to the more capable model. Saving money only matters if the outcome is still acceptable. A failed or low-quality implementation wastes more time -- and therefore more money -- than the heavier model would have cost. Recommend a lighter model only when you are confident the brief is detailed enough for it to succeed.
Management session review checklist¶
Management session review checklist (shared block; do not edit --
the canonical copy lives in shakenfist/development at
templates/shared-blocks/plan-review-checklist.md):
After a sub-agent completes, the management session verifies:
- The files that were supposed to change actually changed -- read them, do not trust the summary.
- No unrelated files were modified.
- The changes match the intent of the brief: not merely syntactically correct, but semantically right.
- The project's own pre-merge checks pass, including any generated code that has to be regenerated and committed (see the project-specific checks below).
- The commit message follows project conventions, including
the
Co-Authored-Byline recording model, context window, and effort level.
In this project
The project-specific checks referred to above are:
-
make instarbuilds andmake lintis clean. - Guest binaries pass
make check-binary-sizes(768KB limit per operation). -
make test-rustand the relevantmake test-integrationtargets pass. -
pre-commit run --all-filespasses.
Administration and logistics¶
Success criteria¶
We will know when this plan has been successfully implemented because the following statements will be true:
make instarbuilds andmake lintis clean.- Guest binaries pass
make check-binary-sizes(768KB limit). - All Rust unit tests pass (
make test-rust). - All Python integration tests pass (
make test-integration). pre-commit run --all-filespasses.cargo tree -p instargains no HTTP, TLS or async-runtime dependency, and the generated.deband.rpmdependency lists gain no new shared library.- The fetcher is started before the VMM opens
/dev/kvmand holds no descriptor but its socket, asserted by a test that reads/proc/<pid>/fd. - Confinement is applied, and whatever the running kernel refuses is reported rather than silently skipped.
- A read-only operation on an
https://input produces byte-identical output to the same operation on the downloaded file, across the read-side format roster. - A sparse source transfers materially less than its virtual size, measured rather than asserted.
- A URL is refused by name for mutating operations, for outputs, and for image-supplied backing-file references.
- The VMM's framing parser has a
cargo-fuzztarget. - Documentation in
docs/describes the feature, anddocs/security.mdstates what the partition does and does not cover. ARCHITECTURE.md,README.md,AGENTS.md, andCHANGELOG.mdhave been updated as needed.
Documentation index maintenance¶
When creating a new master plan from this template, update
the following files in docs/plans/:
index.md— add a row to the Master plans table with the creation date, a link to the plan, a one-line intent summary, the initial status, and links to each phase plan file. Keep the table in chronological order.order.yml— add an entry for the new master plan so it appears in the documentation navigation bar. Phase files should not be added toorder.yml.
When all phases of a plan are complete, update the status
column in index.md to Complete.
Plan close-out sections (shared block; do not edit -- the
canonical copy lives in shakenfist/development at
templates/shared-blocks/plan-closeout-sections.md):
Future work¶
We should list obvious extensions, known issues, unrelated bugs we encountered, and anything else we should one day do but have chosen to defer to here, so that we do not forget them.
- Queue depth greater than one. The real fix for serialised latency is for the guest to have several reads outstanding, which would speed up local operation too. It is much wider than this plan -- it changes the call table and every operation's read loop -- and this plan works around it with host-side readahead instead.
- The guest's wait is a busy poll, and phase 1 made its cost
unbounded.
do_requestspins oncore::hint::spin_loop()with no yield, so a host read that takes N seconds now burns a vCPU core for N seconds where the old spin budget capped it at about 1.3 s before failing. That is a non-issue against a local disk and exactly the case this plan creates: a remote backing store with multi-second cache misses and roughly 164,000 serialised round trips would spin at 100% for the whole transfer. Making the wait yield -- anhltwoken by the completion interrupt, or a notification MMIO exit -- belongs with phase 3 or phase 6, where the latency arrives, and it pairs with the queue-depth entry above because both are changes to how the guest waits. Raised as aconsideritem on the phase 1 review. - Remote output, which needs a writer that does not seek backwards, or a staging file, and is a different feature.
- Authentication, deliberately excluded by open question 7. If it ever arrives, the credential belongs in the fetcher and never in the VMM, and the error paths from phase 8 have to be revisited.
- Remote backing chains, which would need a URL allowlist analogous to the existing path allowlist, and a decision that open question 9 currently answers with "never".
- The FUSE deployment pattern. An unprivileged HTTP-to-FUSE
mount (
httpdirfs,rclone mount, or a smallinstar-httpfs) gives the same partition with no change toBackingStoreat all, and the kernel page cache does the caching and readahead. It is worth documenting as a zero-code alternative even though/dev/fuseis not reliably available in containers. Phase 1 is a prerequisite for it too, since FUSE read latency also blocks the VMM's I/O thread. nbdkitas an alternative front end, which would deliver most of this feature today at the cost of speaking a negotiated protocol in the privileged process.- Issue #860, the 512-byte-per-request
createpath, which phase 6 must coalesce around and which would be better fixed at source. - Caching across invocations, so that chain discovery and a following convert do not refetch the same headers in separate runs.
- The VMM installs no logger unless
--verboseis passed (main.rs's singleinit_verbose_loggercall), so every diagnostic record in the binary is discarded by default, not only guest I/O errors. Wider than #924 was, and filed as #931, which weighs always installing the logger at a lower level against documenting the current behaviour as deliberate. copyreports a failed read as success. Everysend_errorpath insrc/operations/copy/src/main.rsreturns before the singlesend_complete(..., true)at the end, and the host'srun_copyneither requires a completion nor inspects one, so a failed read gives exit 0, no message outside--verbose, and a silently short output file. Filed as #930; the host half is the one that matters, sincerun_copyshould require a completion the way the other operations' hosts do.- Carrying the virtio status byte into the guest's I/O helpers.
do_requestreduces it tostatus == VIRTIO_BLK_S_OK, andread_sector,write_sectorandflushreturnbool, soshared::IoFailure'sstatusfield is unused. Widening those three signatures would let the guest say why. - Issue #849 (convert and dd keep their output file when the guest fails) stays out of scope. The phase 1 plan's Measurements section records that qemu-img keeps its own output on the same failure, which is evidence for whoever takes it.
- Issue #932, a virtqueue error the host cannot read, which is
never completed and so hangs the guest now that the wait is
unbounded. Every trigger is an instar bug rather than image-supplied
data, and the fix is shared fatal state plus a way to kick the vCPU
out of
KVM_RUN, so phase 1 documented the case indocs/guest-architecture.mdinstead of fixing it. It is the one real cost of removing the spin budget. - Issue #933,
docs/testing.md's test-structure list naming 11 of 35 test modules. Phase 1 added its own two and the shared rig, and marked the list as curated rather than complete; completing it, with a CI guard intools/ci/check-test-partition.py, is the fix.
Bugs fixed during this work¶
This section should list any bugs we encounter during development that we fixed. You should also scan the project's issue tracker, where one exists, for directly related issues that we should either resolve as part of this master plan or at least be aware of while planning it.
- The guest spin budget, issue #923, reproduced on 2026-11-18 (open question 5): one host read slower than about a second failed the operation. Fixed in phase 1: the guest now waits for the host to complete the request.
- convert's reporting of an I/O failure, issue #924, from the
same reproduction: a bare "convert operation failed" at exit 1,
with no mention of I/O unless
--verbosewas passed. Fixed in phase 1:convertandddnow name the image, the offset in decimal and hex, and the errno, at exit 1. The truncated output file left behind is #849, not part of #924. - The LUKS inner-format probe, issue #926. The
&&-chainedread_input_sectorcall inconvert_native_lukswas not a silent-zero-fill bug but a transient-failure misdetection: a failed read of the first payload sector looked like "no QCOW2 magic", so a LUKS-wrapped QCOW2 was converted as a flat payload at exit 0. Fixed in phase 1: a failed read of a sector that exists now fails the operation. - A host-side hang, found and fixed with no issue of its own.
BlockDevice::do_readpropagated a backing-store failure without completing the virtio request, so once the guest's wait became unbounded an ordinaryEIOhung it forever. A failed request is now completed withrequest_status::IOERR. This commit and the guest wait change are a pair for a bisect: between them, one host read error hangsconvert. - Issue #860 (
createissues one 512-byte virtio request per sector) is adjacent rather than in scope: it makes the remote case pathological, and phase 6 has to coalesce regardless.
No open issue tracks remote or streaming input; a search of the tracker at planning time found none.
Back brief¶
Before executing any step of this plan, please back brief the operator as to your understanding of the plan and how the work you intend to do aligns with that plan.