Fetch Protocol Crate¶
The fetch-protocol crate defines the channel between instar's privileged
host process and a separate, untrusted fetcher process that serves image
bytes to it.
Overview¶
The host process holds /dev/kvm and the user's credentials. The fetcher
holds whatever it takes to produce bytes, and is assumed hostile. The two are
joined by one AF_UNIX / SOCK_SEQPACKET socket and nothing else: no shared
memory, no filesystem path, no serialisation library. The crate is designed
for:
- A small parse surface: the privileged side parses four fixed-size headers followed by opaque bytes, and nothing more
- Bounds checking before reading: every decode checks the datagram's length against the header's own size before it reads a single field
- Offset-addressed reads: capacity is discovered at init, then the host
asks for
lengthbytes atoffset - A closed error vocabulary: a failure is a number from a fixed set, never text from the peer
Today the only thing that speaks this protocol is instar-fetch, a fetcher
that serves one already-open file. Nothing in the instar binary invokes it
yet.
Location¶
crates/fetch-protocol/
├── Cargo.toml
└── src/
├── lib.rs # Crate overview and re-exports
├── wire.rs # The four headers, Status, block-size negotiation
├── socket.rs # The SOCK_SEQPACKET socketpair wrapper
└── client.rs # FetchClient: the host side's reader
fetch/ # The instar-fetch binary (package instar-fetch)
├── Cargo.toml
└── src/
└── main.rs
Wire Format¶
Every multi-byte field is little-endian, written with to_le_bytes and
read with from_le_bytes. Each header is #[repr(C)] with no implicit
padding. No datagram is ever reinterpreted as one of these structs: decoding
reads one field at a time and validates as it goes, which is what lets the
status field be a real enum rather than an arbitrary integer.
| Message | Size | Fields (offset in bytes) |
|---|---|---|
| InitRequest | 16 | magic 0, version 4, max_datagram 8, flags 12 |
| InitReply | 32 | magic 0, version 4, status 8, block_size 12, capacity 16, flags 24 |
| ReadRequest | 32 | opcode 0, length 4, id 8, offset 16, flags 24 |
| ReadReply | 24 | opcode 0, status 4, id 8, length 16, flags 20 |
All fields are u32 except capacity, flags (InitReply and ReadRequest),
id and offset, which are u64.
The layout tests in wire.rs are the authority. They assert size_of
and the offset of every field of every header against these numbers, so a
reordering that would change the wire format breaks a test. If this table and
those tests ever disagree, the tests are right.
Further rules that apply to every message:
- The magic is the little-endian encoding of the bytes
INFT, so that it reads as text in a hexdump of the datagram. flagsis reserved and must be zero in every message. A peer that sets a bit this build does not know is refused rather than guessed at.- Trailing bytes are an error. InitRequest, InitReply and ReadRequest are
exactly their header length. A ReadReply is its 24-byte header followed by
exactly
lengthbytes of payload in the same datagram.
Message Types¶
Host to fetcher¶
| Message | Purpose | Fields |
|---|---|---|
| InitRequest | First datagram; says who is speaking and what the host can carry | magic, version, max_datagram, flags |
| ReadRequest | Asks for a range of the source | opcode, length, id, offset, flags |
Fetcher to host¶
| Message | Purpose | Fields |
|---|---|---|
| InitReply | Accepts or refuses the init; settles block size and reports capacity | magic, version, status, block_size, capacity, flags |
| ReadReply | Answers one request, matched by id | opcode, status, id, length, flags, then payload |
The only opcode is 1 (read), used in both ReadRequest and ReadReply. Values
2 and 3 are reserved for a future flush and a capacity re-query; a request
carrying either is refused today.
Status vocabulary¶
The status is a closed set owned by the privileged side. It is rich enough to act on, and it cannot carry a payload: the sentence a user reads is written by the host, from a number.
| Value | Name | Meaning |
|---|---|---|
| 0 | Ok | The request was satisfied in full |
| 1 | BadRequest | The request was malformed: a reserved field set, a length past the negotiated maximum, an offset and length that overflow |
| 2 | Unsupported | Well formed, but asks for something the fetcher does not implement, such as an unknown opcode |
| 3 | OutOfRange | The range lies outside the capacity the fetcher advertised |
| 4 | ShortRead | The source ended before the requested length; the reply carries the bytes that did exist |
| 5 | IoError | Reading the source failed |
| 6 to 11 | reserved | Documented, not defined |
Values 6 to 11 are reserved, in order, for NOT_FOUND, FORBIDDEN,
RANGE_UNSUPPORTED, SOURCE_CHANGED, NETWORK and TIMEOUT, for a fetcher
that speaks to something further away than a file. They are refused today:
a peer sending 6 to 11 is treated exactly like one sending any other unknown
value, as a protocol error. Adding one is a deliberate change to the Status
enum and to the sentence that renders it.
The client maps each non-Ok status to an io::ErrorKind chosen by this crate:
BadRequest and OutOfRange to InvalidInput, Unsupported to Unsupported,
ShortRead to UnexpectedEof, IoError to Other. A reply that cannot be
decoded, or that names an unknown status or id, is InvalidData.
Init Negotiation¶
The host sends an InitRequest first. It carries the protocol version and
max_datagram, the largest single datagram the host's end can send or
receive. The fetcher replies with an InitReply carrying its own version, a
status, the settled block_size (the most payload one ReadReply may carry),
and capacity, the size of the source in bytes.
The version must match exactly; a mismatch is fatal and names both versions, because a development build talking to a packaged fetcher is a mistake that will happen. The magic is checked on both init messages.
What a side can offer comes from the socket buffer the kernel actually
granted, not from a constant or a sysctl. Creating a socket pair asks the
kernel for buffers large enough to carry the largest block the protocol
permits, on both ends, and the fetcher asks again for its own end in case it
was handed a socket it did not create. Each side then reads back what it was
given. The kernel clamps a request to wmem_max and then doubles it, and
charges 32 bytes of overhead per datagram, so the largest datagram a side can
manage is granted - 32 -- a figure that cannot be derived from the sysctl by
inspection, which is why it is measured rather than assumed. A caller that
wants a smaller buffer shrinks it after creating the pair, and nothing puts it
back.
The fetcher then settles the block size:
- take the smaller of the two sides' maximum datagram sizes,
- deduct the 24-byte ReadReply header,
- cap the result at the 65536-byte default block, and
- round down to a multiple of 512 so a split read stays sector-aligned.
Anything below 512 bytes is refused rather than used.
The cap is the default block rather than the 262144-byte maximum the wire format allows, so a well-provisioned pair settles on 65536 rather than on the largest block it could carry. A larger block buys nothing while a guest's own request is at most that size: it would only change how a larger read is divided, and would hold a receive buffer four times the size for payloads that never arrive. Reading ahead is a matter of several requests being in flight, not of one request being bigger. The 262144-byte maximum is what the headers and the requested socket buffers are built to reach, so raising the default later is a change to one constant rather than to the protocol.
As a measured example: on the host this was developed on, a pair is granted 425984 bytes an end, which is a 425952-byte datagram ceiling -- comfortably more than the default block needs, so the negotiation settles on 65536. A socket deliberately shrunk to ask for only 4096 is granted 8192, which is an 8160-byte ceiling and a 7680-byte block, and reads larger than that are split. The client accepts the fetcher's proposed block only if it satisfies every rule the negotiation itself follows: at least 512, a multiple of 512, and no larger than what this side could carry -- which, being capped at the default block, is the figure a refusal names. The alignment is checked rather than assumed, because it is what keeps a split read on sector boundaries and the fetcher is the untrusted side: a peer proposing 513 would otherwise impose both a misaligned split and a round trip for every 513 bytes.
A read larger than the negotiated block is split by the client into several
requests. The timeout bounds the whole read_at, not each datagram. A
split read is several round trips, and a peer answering each one just inside
a per-datagram timeout could hold the privileged process indefinitely
without ever tripping it, which is the hang the partition exists to prevent.
Each block is given what remains of the call's deadline, and a deadline that
runs out is TimedOut naming the offset the peer stopped at.
The deadline covers both directions of each round trip. A peer that stops
answering blocks the reply, and a peer that stops reading blocks the request
itself once its receive buffer is full, which is the same hang reached from
the other end; SO_SNDTIMEO and SO_RCVTIMEO are both set from what remains
of the deadline. Below one microsecond a timeval rounds to zero, which the
kernel reads as no timeout at all, so the socket refuses such a value rather
than quietly removing the bound -- and the client refuses a whole-read
deadline shorter than the window one round trip is given, which would fail
every read before sending anything.
Framing¶
SOCK_SEQPACKET preserves message boundaries in the kernel, so there is no
length prefix: one recv is one message, or an error. The host never has to
reassemble a stream or recover from a framing desynchronisation, which is the
failure a byte-stream channel would add to a protocol whose peer is hostile.
Rust's standard library cannot create such a socket. UnixStream's
constructors hardcode SOCK_STREAM, UnixDatagram's hardcode SOCK_DGRAM,
and nothing takes a socket type. The choice was a third-party socket crate in
the process that holds /dev/kvm, or a few libc calls behind a type that
owns its descriptor. This crate does the latter: SeqPacketSocket::pair() is
a libc::socketpair(AF_UNIX, SOCK_SEQPACKET | SOCK_CLOEXEC, 0) behind a safe
wrapper that closes on drop, in the same shape as the VMM's ioeventfd
wrapper.
It is a socketpair rather than a socket bound to a path because the host is the fetcher's parent: there is no name for anything else to connect to, no directory to choose, and nothing for a filesystem sandbox to have to permit.
No Free-Form Text¶
There is no free-form text anywhere on the wire. A failure is a value from
Status, and the sentence a user eventually reads is written by the host from
that value. A reason string from the untrusted side would be
attacker-influenced text heading for a terminal, and would need sanitising,
length bounds and encoding validation in the one process that can least afford
a parsing mistake. Every ProtocolError carries numbers this side computed or
read, never bytes from the peer. The fetcher's own stderr remains available
for diagnostics that never become part of the protocol.
That stderr carries one line per refused or non-Ok request, so its volume
is set by the peer rather than by the fetcher. Nothing reads it today -- the
only peer is the privileged side itself -- but whatever eventually routes it
into a log owns that rate, and should bound or sample it rather than assume
the fetcher is quiet.
Usage¶
Host side¶
use fetch_protocol::{FetchClient, SeqPacketSocket};
let (host_end, fetcher_end) = SeqPacketSocket::pair()?;
// ... hand fetcher_end and an open file to instar-fetch as descriptor numbers ...
let mut client = FetchClient::new(host_end)?; // performs the init exchange
let mut buf = vec![0u8; 4096];
client.read_at(0, &mut buf)?; // fills buf or returns an error
read_at has the same signature and contract as the VMM's local backing
store: fill the whole buffer or return an error. That includes the end of the
source. A block device's capacity is its image's length rounded up to a whole
sector, so a guest reading the last sector of an image that is not a sector
multiple legitimately reads past the end of the file; both stores answer the
overhang with zeros, and the client produces those zeros itself rather than
asking the fetcher for bytes it would refuse as OutOfRange. A read starting
past the capacity is zeros and no round trip at all.
The capacity is the size the fetcher advertised at init, and it is not
re-read. A source that grows afterwards therefore reads back as zeros past
the advertised size rather than as the new bytes, and a source that shrinks
fails with UnexpectedEof from the fetcher's ShortRead. Both are what the
VMM's local backing store does with a file that changes under it, which is
the point: the two stores answer a changing source alike, so whatever sits
above them cannot tell which one it has. The client matches each
reply to its request by id rather than assuming order, and a reply for an id
it did not ask for is an error. The first failure ends the client, because the socket
cannot be resynchronised without knowing how many answers are outstanding.
Fetcher side¶
Both arguments are descriptor numbers the parent has already opened. The
fetcher takes no paths and opens nothing, so it can in principle be given no
filesystem access at all. It is single-threaded, answers each read with
pread on the file, and never panics on anything the peer sends: a malformed
request is answered with a status, or noted on stderr. It exits 0 when the
peer closes the socket and for --help, 2 for a usage error, and 1 for
anything else -- including a descriptor that is not a regular file, which is
refused before the init reply rather than failing every request.
A datagram longer than the fetcher's request buffer is refused by its id like
any other malformed request. MSG_TRUNC reports the length that was really
sent while the kernel keeps only the head of it, and the head is where the id
lives, so the remainder being gone is no reason to leave the peer without an
answer.
One malformed shape is answered with silence: a datagram too short to carry a request id. A reply is addressed by the id it carries back, so there is nothing to address, and an invented id could be matched by a peer that had lost framing to a request that really is outstanding. The client's read deadline is what covers it, and no correct client can produce the shape, since the request header is fixed-size.
Dependencies¶
libc: thesocketpairandsetsockoptcalls. This is the crate's only dependency.
Current Status¶
The only thing that speaks this protocol today is instar-fetch serving one
open file. It is built and packaged, but nothing in the instar binary
starts it or links the client. Tests cover the layout, each malformed shape
decoding to a typed error, an end-to-end comparison of the fetcher against
pread, a set of hostile peers, and a fuzz target on every decode entry
point.
Limitations¶
- One request in flight. Replies are matched by id, so the protocol permits out-of-order replies, but the client keeps one request outstanding.
- Read only. The flush and capacity re-query opcodes are reserved, not implemented.
- Capacity is fixed at init. There is no way to learn that the source changed size.
- Block size is capped at 64 KiB by the negotiation, however much socket buffer the kernel grants, and at 256 KiB by the wire format. Raising the first is a change to one constant; raising the second is a change to the protocol.
Why Not NBD, and Why Not a Virtqueue¶
The model is borrowed from virtio-blk and NBD: capacity discovered at init, offset-addressed reads, a typed status, request ids. Neither mechanism is.
Not a virtqueue. The repository already depends on virtio-queue and
vm-memory and has eventfd notification, and SCM_RIGHTS could pass a memfd
over the socket. It would work, but it would hand the untrusted fetcher a
writable mapping shared with the privileged process, where whoever controls
the ring controls indices the VMM dereferences. That is the memory-corruption
channel this partition exists to remove, and defending it means writing
adversarial-ring validation in privileged code to save a copy that does not
matter: a 256 KiB memcpy costs tens of microseconds.
Not NBD. NBD has exactly this shape, and nbdkit is an off-the-shelf
implementation of much of it. But speaking real NBD means parsing a negotiated,
extensible protocol (option haggling, structured replies) from an untrusted
peer in the privileged process. A few fixed-size headers are a far smaller
surface, small enough to fuzz the whole decoder. The shape is borrowed; the
protocol is not linked.
Not protobuf, unlike the guest serial channel. guest-protocol uses protobuf because guest results are variable-shape messages with optional fields. This channel carries fixed-size headers and raw bytes, so a serialisation library would add a dependency and a decoder with nothing to offer in return.