Kerbside Proxy Architecture

This document provides a detailed technical description of Kerbside's proxy architecture: the process model, the connection state machine, traffic handling, and the SPICE firewall.

The SPICE proxy is the Rust kerbside-proxy binary (rust/kerbside-proxy/). The pure-Python kerbside package provides the REST API, the console-source drivers, the data model, and the daemon that supervises the proxy; the proxy itself terminates TLS, drives the SPICE handshake, and relays traffic, consulting the Python side over a gRPC control socket for authorization and bookkeeping. It reuses the ryll shakenfist-spice-protocol crate for the SPICE wire format.

Why the proxy has this shape

• Rust, and policy left in Python. The proxy parses bytes from the internet and passes them to qemu's spice-server, which its upstream says should not face the internet and whose demarshalling of client messages has a history of memory-safety CVEs. The Python proxy this replaced relayed opaquely once a client had authenticated, and ran a process per channel. Rust brings memory safety to the component that parses untrusted input, and a task rather than a process per connection; the Python side keeps the database, the API and every policy decision ("Python decides policy; Rust enforces it").
• gRPC over a unix socket for the control plane. It brings message framing, request/response correlation, deadlines and server push (the ProxyControl stream). Shaken Fist's privexec/nodelock pattern, bare protobuf over a unix socket with no message framing, was rejected because a Rust peer cannot speak it reliably; Shaken Fist tracks it as technical debt in shakenfist/shakenfist#3315, and its own database daemon already uses gRPC.
• Framing kept even where policy passes everything. The relay frames the bulk server→client direction too, rather than splicing it with copy_bidirectional: recording a session, rewriting a message and the proxy originating traffic of its own (the danger-zone recording indicator, a Kerbside-provided USB device) are not built, but all of them need message boundaries, and an unframed fast path would rule them out. The client→server direction, where strict parsing matters, carries little traffic, so parsing it is cheap.

Process Architecture

kerbside daemon run (kerbside/main.py) supervises the proxy as a single child process (subprocess.Popen, via kerbside/proxy_supervisor.py), binding the gRPC control socket first. The proxy is an async tokio program: it accepts connections and handles each on its own task, rather than forking a worker process per connection. Shared state (tokens, channel bookkeeping, audit) lives in the database, reached indirectly through the control-plane gRPC service — the proxy never touches MariaDB directly.

flowchart TD
    daemon["kerbside-daemon<br/>binds the gRPC UDS,<br/>then supervises the child"]
    proxy["kerbside-proxy (Rust)<br/>one tokio task per connection"]
    service["KerbsideProxy service<br/>(in the daemon)"]
    hypervisor["hypervisor SPICE"]

    daemon -- "subprocess.Popen" --> proxy
    proxy <-- "gRPC over UDS:<br/>authorize, audit, channel records" --> service
    proxy -- "TLS relay" --> hypervisor

Connection Listener

listen.rs binds two ports (defaults):

• Secure (5900): TLS. The proxy terminates client TLS with its host certificate, then drives the SPICE link handshake.
• Insecure (5901): a minimal responder that issues the SPICE need_secured redirect, steering clients to the TLS port. No plaintext SPICE traffic is proxied, but clients that dial port= first still need to reach it; see network-ports.md.

Accepted client sockets get TCP keepalive (mirroring the backend leg), so a silently dead peer is detected rather than pinning a task forever.

TLS Configuration

The proxy holds a host certificate/key (PROXY_HOST_CERT_PATH / PROXY_HOST_CERT_KEY_PATH) and a CA bundle (CACERT_PATH). Client-facing TLS is terminated with the host certificate. The backend leg (backend.rs, via the ryll SpiceClient) originates TLS to the hypervisor. When the console carries a host_subject, the backend leg verifies the server certificate's subject against it using spice-common matching semantics, substituting for hostname verification, so a mis-issued or substituted backend certificate is rejected. When the console has no host_subject, the CA chain is the only identity check. The rustls ring crypto provider is installed at startup.

A console whose Target carries a tunnel takes a different route. The proxy reaches qemu's TLS port only through the HTTP CONNECT proxy named by tunnel.proxy_uri, asking it for tunnel.connect_host, and dials nothing else: neither hypervisor_ip nor insecure_port is touched. There is one TLS attempt and no plaintext attempt or need_secured retry. A tunnel needs a host_subject; the ryll client refuses one without it, and the refusal is recorded as a failed connect. An empty connect host, an unusable secure_port or a proxy URI ryll cannot parse is refused before any dial. The connect host is a credential, so the proxy never logs it and its errors name the field, not the value; its log lines say only whether the target was tunnelled.

The console's ca_cert is the only trust anchor for the backend leg when it is set: the public web roots are not consulted, so a hypervisor certificate must chain to that CA and nothing else will do. Without a ca_cert the public web trust store applies instead, and then the certificate's name must match the host dialled unless a host_subject is pinned. This is narrower than earlier releases, where a configured ca_cert was added to the public roots rather than replacing them, so a certificate from any public CA also passed the chain check (ryll commit 3050082, picked up with the fix for issue #477).

What that means depends on the source type, because ca_cert is not always a free choice:

• Shaken Fist and oVirt. One ca_cert field serves both the cloud API and the hypervisor VDI leg, and the source driver requires it to match the CA the cloud itself publishes, exactly, or the source is marked errored (kerbside/sources/shakenfist.py, kerbside/sources/ovirt.py). So it can be neither left empty nor extended into a bundle. Both platforms issue hypervisor SPICE certificates from that same cluster (or engine) CA, so the narrower trust changes nothing for them. A hypervisor whose SPICE certificate was replaced with one from another CA will now fail the backend connect.
• Proxmox VE. ca_cert is required and is the cluster CA, which verifies the API and is also the trust anchor for the backend leg, since that CA signs each node's SPICE certificate. It is not compared with anything the API reports, and the ca in a minted ticket is ignored.
• Static and OpenStack. ca_cert is only the VDI trust anchor (see Console Sources). A deployment whose hypervisors present publicly issued certificates must leave it empty for that source rather than set it to an unrelated CA.

For Shaken Fist consoles the host_subject is not configured on the proxy: it is taken at scrape time from the hosting node's published SPICE server certificate subject (spice_server_cert_subject), and carried through to the proxy in the AuthorizeConnection Target. That pin is conditional. A node that publishes no subject (an older cluster, or a node without a SPICE certificate) yields no host_subject, and the backend leg for its consoles is then relayed without host-subject enforcement rather than refused (backend.rs maps an empty subject to none). The per-source synthesize_host_subject option substitutes CN=<node fqdn> in that case, which is only correct on the stock cn=hostname PKI and so is off by default. See Console Sources.

Backend host subject pinning

The matching is done by the TLS verifier in ryll's shakenfist-spice-protocol crate, whose host_subject.rs module comment states the grammar and comparison rules. The proxy only passes the console's host_subject through (build_config in backend.rs). Its shape has these reasons:

• The rules are spice-common's, not the earlier Python proxy's. They replicate subject_to_x509_name and verify_subject in spice-common's common/ssl_verify.c, which spice-gtk and virt-viewer apply to a .vv file's host-subject, so one subject string is accepted or refused the same way by virt-viewer, ryll and this proxy. The Python proxy rebuilt a string from only the C, O and CN attributes and compared it exactly, so a certificate carrying any other attribute could never match. Two differences from OpenSSL are deliberate: values are compared with ASCII case folding only, and the attributes of a multi-valued RDN are compared in the order the certificate holds them rather than sorted. Either can refuse a certificate OpenSSL would accept, never the reverse.
• It fails closed. An unknown attribute key, a malformed string, a value encoding the matcher cannot decode (TeletexString, BMPString) and a mismatch all refuse the backend. A malformed pin is rejected when the ryll client is constructed rather than at the handshake, so it fails even where the plaintext attempt would have succeeded, and never quietly becomes an unpinned connection.
• A pin replaces hostname verification, as it does in spice-gtk, because SPICE server certificates usually carry no subjectAltName.
• A refusal is audited as an ordinary failed connect. The audit event is Hypervisor connection failed: followed by the error chain, which for a mismatch ends invalid peer certificate: NotValidForName; the proxy's log line TLS: rejecting certificate: pinned host_subject ... names both subjects. There is no separate audit event type, as there was none in the Python proxy.
• oVirt's subjects need no normalisation. The oVirt API reports host.certificate.subject already in the spice-common form (O=local,CN=ovirt.local from the oVirt CI engine, 2026-07-17), so the oVirt source stores it as reported.

The direct-qemu lane proves both outcomes against a real TLS backend; see Backend host subject checks.

Connection State Machine

Each accepted connection runs this sequence (session.rs, backend.rs, relay.rs):

1. TLS terminate the client connection (secure port).
2. SPICE link handshake using the ryll server-role handshake drivers: parse the client SpiceLinkMess and reply. A client whose common capabilities lack MINI_HEADER or AUTH_SELECTION is refused here with a version_mismatch link error, since the relay frames on the mini header and the ticket read expects an auth selector. Otherwise the reply offers a fixed set of capabilities per channel type (see "Link capabilities" below), and the client's own capabilities are kept for the backend leg.
3. Ticket decryption / authorization. The client's RSA-encrypted ticket is decrypted and sent to the control plane via the AuthorizeConnection RPC, which resolves it to a hypervisor Target (host, port, backend ticket, host_subject, session_id, and the firewall policy) or returns Denied. A denial closes the connection. For a source whose driver mints tickets (oVirt and Proxmox VE), the control plane may call that source's API while authorizing (see "Minting a backend ticket during authorization" below), and denies the channel if it cannot.
4. Backend connect. The proxy opens the backend leg to the hypervisor's SPICE port (honouring a need_secured retry to the TLS port) and completes the server-side handshake with the backend ticket, advertising the client's capabilities rather than the proxy's own. For a tunnelled target it instead makes a single TLS connection through the target's HTTP CONNECT proxy (see "TLS Configuration").
5. Relay. Traffic is relayed in both directions until either side closes, the idle-read timeout fires, the firewall issues a terminating verdict, or the control plane terminates the session.

The connection registers in the session registry (session.rs) under its session_id after authorization and deregisters on teardown, so the control plane can drop it (see "Session termination" below).

Link capabilities

The client leg's link reply has to be sent before the backend is known: it carries the RSA key the client encrypts its ticket to, and the ticket is what selects the backend. So the capabilities Kerbside offers a client cannot depend on the backend. Instead, for each channel type it offers what spice-server itself advertises for that type (rust/kerbside-proxy/src/caps.rs cites the spice-server source for each):

Channel	Offered channel capabilities
common (every channel)	AUTH_SELECTION, AUTH_SPICE, MINI_HEADER (never AUTH_SASL)
main	SEMI_SEAMLESS_MIGRATE, NAME_AND_UUID, AGENT_CONNECTED_TOKENS, SEAMLESS_MIGRATE
display	MONITORS_CONFIG, STREAM_REPORT, PREF_COMPRESSION, PREF_VIDEO_CODEC_TYPE
inputs	KEY_SCANCODE
playback	VOLUME, OPUS
record	VOLUME, OPUS
usbredir, port, webdav	DATA_COMPRESS_LZ4
cursor, smartcard	none

Every message those capabilities let a client send is in the L1 allowlist.

The backend leg then advertises the client's capabilities, verbatim and including any extra capability words, so spice-server chooses image compression, stream codecs and drawing operations for the client actually connected. The one exception is the authentication mechanism bits, which describe Kerbside's own ticket exchange with the backend: AUTH_SPICE is set and AUTH_SASL cleared. Before issue #477 was fixed the backend leg always advertised ryll's capabilities, so spice-server could, for example, send H.264 streams to a client that had not offered H.264.

If a backend grants fewer channel capabilities than Kerbside offered the client (a spice-server built without Opus or LZ4, say), the proxy logs a warning naming the missing capability bits; the backend's reply is logged at debug level. The client may then send something that backend refuses, which cannot be undone once the client leg's reply has been sent.

Packet Framing and the Relay

Post-handshake SPICE traffic uses a 6-byte mini-header (MessageHeader: a u16 message type and a u32 size) ahead of each body. The relay (relay.rs) frames every message by that header rather than copying bytes opaquely, which is what makes inspection and the firewall possible. Each direction is a pump; tokio::select! drives both plus a cancellation arm, so whichever future finishes (a closed half, a timeout, a terminating verdict, or a control-plane cancellation) tears the whole connection down cleanly.

Before a body is buffered, an L0 check bounds it (per-channel size caps and an optional rate ceiling), so an attacker-controlled length in the header cannot drive an unbounded allocation. Framed messages are then checked at L1 against the per-channel/direction message-type allowlist. See "the SPICE firewall" below.

Channels

The proxy models the standard SPICE channels: main, display, inputs, cursor, playback/record, usbredir, and the port channel. The L1 allowlist (rust/kerbside-proxy/src/allowlist.rs) is derived per channel and direction from the ryll shakenfist-spice-protocol message-type tables.

Transcoding the display channel (experimental)

By default the proxy relays every channel. An experimental alternative terminates the display channel instead: the proxy decodes what the hypervisor sends and re-encodes it for the client, sending only what changed, as JPEG where the picture is photographic and losslessly otherwise, at the pace the client's link can take. It exists to measure whether that beats relaying on slow links; it is not for production, and the spike's verdict recommends against productising it.

It needs two things, both off by default:

• The transcode cargo feature. It pulls in ryll's renderer and the compression crate's image encoders, whose encode feature also builds openh264 from source. Release wheels never enable it (tools/build-proxy-wheel.sh builds with default features), so a released kerbside-proxy has neither the flags below nor openh264. Build it with make -C rust/kerbside-proxy build FEATURES=transcode.
• --transcode-display. A process-wide proxy flag: every session's first display channel is transcoded. The daemon does not pass it, so run such a build directly (see the direct-qemu harness).

Flag	Default	Meaning
--transcode-display	off	Transcode each session's first display channel.
--transcode-jpeg-quality	85	JPEG quality (1 to 100) for areas judged photographic; 0 sends everything losslessly.
--transcode-lossless-delay-ms	500	How long an area sent as JPEG must go without change before it is sent again losslessly.
--transcode-notsent-lowat-bytes	65536	TCP_NOTSENT_LOWAT on the transcoded channel's client socket: how far ahead of the link the transcoder may write.
--transcode-ack-window	0	A measurement option, hidden from --help: pace on a SPICE ACK window of this many messages, as spice-server does, instead of on the client socket. 0 paces on the socket.
--transcode-switch-after-secs	0	An experiment, hidden from --help: move each session's first display channel between relaying and transcoding this many seconds after it starts, and again each time that long has passed in the new mode. It starts transcoded with --transcode-display and relayed without. 0 never switches. See switching a display channel.

How a transcoded display channel works (rust/kerbside-proxy/src/transcode/):

• Backend leg. The display channel is dialled with fixed capabilities that describe ryll's renderer rather than the client: LZ4 and a compression preference, so draws stay lossless and never become QUIC (which spice-server turns into JPEG on a link its bandwidth test calls slow); MULTI_CODEC with no codec bits, so spice-server creates no video stream; and nothing the renderer cannot draw. ryll's renderer reads the channel, answers spice-server's flow control at LAN speed, and decodes each draw into a mirror of the guest's primary surface that records where each draw landed.
• Client leg. The proxy answers as a display server: it waits for the client's DISPLAY_INIT, then sends SURFACE_CREATE for the primary surface, a full lossless draw, MONITORS_CONFIG if the client has that capability, and MARK. After that each frame is the draws that bring the client up to date, then MARK. A resize from the guest becomes SURFACE_DESTROY and the same start sequence. No video stream is ever sent, and no image asks the client to cache it. The client's messages pass through the same L0 and L1 firewall as the relay's, and its bytes count in kerbside_proxy_bytes_relayed_total. Unlike the relay, the client leg has no idle-read timeout, because once initialised a client paced on the socket has nothing to say. TCP keepalive ends a client that has gone. A client that stays connected but stops reading (a zero window, or no ACKs under ACK pacing) is ended, logged at info, once a write to it has made no progress for 15 minutes, the relay's idle-read timeout. A slow link is not a stall: the bound is on progress, not on how long a frame takes.
• What a frame sends. The surface's damage says where to look; each damaged 64-pixel tile is compared with a shadow of what the client already holds, and only tiles that differ are sent. Tiles are merged into at most 32 rectangles a frame, and a rectangle over 4 MiB raw is split, keeping every message far below the proxy's 16 MiB cap. A rectangle that compresses poorly with LZ4 (more than half its raw size) is tried as JPEG, the rest go as LZ4. JPEG is used only if the client leaves the compression choice to the server. This update pipeline is copied from andris, whose thresholds were measured on real desktop screens.
• Pacing. One frame is written at a time, and the next starts only once the kernel has taken the last, which TCP_NOTSENT_LOWAT holds back until the link has nearly drained it. Meanwhile damage merges in the mirror, so a slow client is sent the screen as it is when it can take more, never the frames it missed. No SET_ACK is sent, so the client sends no ACKs.
• Memory. Each transcoded channel holds ryll's pixmap cache (up to 256 MiB) and GLZ dictionary (up to 64 MiB) besides the mirror and the shadow. When the channel ends the proxy logs, at info, how full the pixmap cache got (pixmap_cache_peak_bytes, pixmap_cache_peak_entries) and how many entries it evicted. The shaped-link rig's summarise.py reports the peak.

What it does not do:

• Only a session's first display channel (channel id 0) is transcoded. A second display channel is relayed, and the proxy logs that it did so.
• The choice between relaying and transcoding is made when the channel connects. Only the hidden --transcode-switch-after-secs experiment changes it while the channel runs: it keeps the client leg, closes the backend display channel and dials it again for the other mode, and keeps GLZ out of the relay, because GLZ does not survive the switch. Both modes dial with the transcoder's capabilities, so its relay assumes a client that takes LZ4 and MONITORS_CONFIG. Switching a display channel has what that costs, and what breaks in each client.
• The backend leg is not firewalled. The server-to-client messages of a transcoded display channel skip the L0 and L1 policy and the relay's 16 MiB framing cap, and go straight to ryll's renderer, which frames them and decompresses guest-influenced LZ4, GLZ, QUIC and JPEG data inside the proxy process. Only the renderer's own limits bound that decode, so a decoder panic or an allocation the guest can drive is reachable from the guest, in the process that carries every other session. Only the client's messages are firewalled.
• Its scope is a virtio-gpu or std-vga guest, whose display reaches spice-server as bitmap draws on the primary surface. A guest running the QXL driver can send draws ryll's renderer does not implement (ROP3, stroke, text and composite draws, palette, surface and JPEG-with-alpha images, and draws on off-screen surfaces). The transcoder logs the first of each kind in a session and counts every one in kerbside_proxy_transcode_unsupported_total{kind}. It cannot fall back to the relay by then, because the client already holds the transcoder's surface, so the client shows a stale region wherever such a draw landed.

Metrics

The proxy exposes a Rust-native Prometheus /metrics endpoint (metrics.rs), bound to --metrics-address (default loopback, config PROMETHEUS_METRICS_ADDRESS) rather than the public VDI address, since the endpoint is unauthenticated. Metrics cover accepted/authorized connections, active connections/channels, bytes relayed per direction, firewall verdicts, spice-server's ACK window, and what a transcoded display channel could not draw. Configuration lists them.

Error Handling

Handshake, TLS, authorization, and backend-connect failures close the connection and are logged with a per-connection correlation id (a tracing span around the connection lifecycle). A denied authorization, a failed backend TLS verification, an oversized/dis-allowed message, and an idle-read timeout each end the connection deterministically rather than leaking a task.

Log Output

The proxy writes tracing output to stdout. Under the daemon that stdout is inherited into the kerbside log, so colouring is decided by the sink: ANSI escapes are emitted only when stdout is a terminal, and a redirected log gets plain text. This matters for anything that parses the log — with colour on, tracing renders a structured field as name<esc>[0m<esc>[2m=<esc>[0mvalue, so a search for a literal name= finds nothing at all.

Security Considerations

Token Validation

Tickets are validated by the control plane, not the proxy: AuthorizeConnection resolves the decrypted ticket against the database and returns a Target or Denied. The proxy enforces the decision and never reaches the database on its own.

Two different tokens exist for a Shaken Fist console, and they live on different surfaces:

• The HTTP JWT bearer is the Ed25519-signed token Shaken Fist mints and a viewer presents at the /sf-console.vv exchange endpoint on the Python API. The API verifies it offline against the cluster's cached signing public keys (signature, aud, exp) and enforces single use through a jti replay table (sf_token_jtis) — a replayed token is rejected. The Rust proxy never sees this JWT; the exchange happens entirely on the Python side, which then issues an ordinary Kerbside console token and .vv file.
• The on-wire SPICE ticket is what the proxy actually authorizes. It arrives RSA-encrypted inside the SPICE link handshake, is decrypted, and is resolved by the control plane via AuthorizeConnection. The proxy enforces only this decision; the JWT exchange has already completed before any SPICE connection is made.

Minting a backend ticket during authorization

For a static source, AuthorizeConnection returns the ticket stored on the console. For a source whose driver sets MINTS_TICKETS (oVirt and Proxmox VE) there is no stored ticket: the first channel of a session to be authorized mints one from the source, and the rest reuse it. The sequence is:

1. The lifetime asked for is the token's remaining life plus 30 seconds (TICKET_SLACK in kerbside/rpc/servicer.py), since no channel is authorized after the token expires. oVirt honours it. Proxmox VE has no way to ask: its tickets always last 30 seconds from the mint, and the reuse window below is what keeps a session's channels on one.
2. db.session_ticket() selects the session's consoletokens row FOR UPDATE. If it holds a ticket with at least 10 seconds left (REUSE_MARGIN) that ticket is returned; otherwise the caller mints and the result is stored. The lock is shared by all proxy nodes, so channels of one session arriving together on different nodes mint once.
3. The mint runs on a worker thread, and the caller waits at most MINT_DEADLINE (20 seconds, kerbside/sources/minting.py). The deadline covers building the driver, logging in and the ticket request, because ovirtsdk4 does not bound its login; each engine request is separately bounded at 15 seconds. A late result from an abandoned worker is discarded, never stored. Mints for one console run one at a time in each daemon: while a worker for the console is still running, even an abandoned one, a new mint waits for it within its own deadline and is refused if it does not finish. Otherwise the late worker could replace the hypervisor's password after a newer mint had set and stored its own, and every later channel would be refused. The Proxmox VE driver narrows that further: once MINT_POST_CUTOFF (10 seconds, kerbside/sources/proxmox.py) has passed since it started, it no longer sends the request that sets the password, so a slow mint gives up before changing it. The cutoff counts from the driver's start, not from when its caller began waiting (#656). A worker still running after five minutes (MINT_STUCK_AFTER) stops holding mints up. Daemons on different nodes do not see each other's workers, nor does the direct .vv handler, which mints in the web workers; and any mint for a VM, whichever session it is for, replaces the password other sessions' stored tickets were for.
4. A failed or timed-out mint stores nothing, denies the channel with console ticket unavailable, and is audited with the exception's class name (never its text, which an API client can fill with the request it failed on). The next channel tries again. A successful mint is audited with its lifetime. No Channel created event is written for a denied channel.

The maintenance loop clears a stored ticket once it or its token has expired. A replacement ticket leaves a session's established channels connected; QEMU's set_password keeps connected clients. Separately, a SPICE server serves one client at a time and drops its connected client when another links, so a second session on the same VM disconnects the first regardless of tickets.

Worst case for the gRPC worker pool. Every channel waiting on the row lock occupies one of the API_GRPC_WORKERS threads (default 8) serving AuthorizeConnection for the whole of the wait. With a healthy engine the wait is one mint. Against an engine that accepts connections and then hangs, the channel holding the lock gives up after MINT_DEADLINE, and each channel behind it then takes its own turn at the lock and its own 20 second wait. A session opening more than a few channels at once can therefore keep that many workers busy for 20 seconds each in turn, and the later ones can wait past MariaDB's lock wait timeout (innodb_lock_wait_timeout, 50 seconds by default); they then fail with an internal error rather than console ticket unavailable, and the client sees the connection refused either way. Sessions on other VMs are not held up by the lock, but all sessions share the worker pool, so a hung engine can exhaust it for every source until the waits end. Each waiting channel also holds a database connection, from the daemon's SQLAlchemy pool (5 plus 10 overflow by default), which the maintenance loop and the other gRPC calls share; raising API_GRPC_WORKERS much above that makes a hung engine exhaust the pool too. The deadline caps each wait; it does not make a hung engine cheap.

Audit Logging

State-changing events (channel created/removed, firewall verdicts, session termination) are recorded as audit events through the control-plane RPCs (RecordAuditEvent, and the channel register/deregister calls), so the audit trail is owned by the Python side and consistent across proxy nodes.

Process and Blast-Radius

The proxy runs as a single unprivileged process consulting a filesystem-guarded UDS (directory 0700, socket 0600); it holds no database credentials and no long-lived secrets. It receives each channel's ticket from the daemon when the channel is authorized, and never logs it. The firewall (below) bounds what an untrusted client can send through to a hypervisor.

The SPICE Firewall

Rather than relaying post-authentication traffic opaquely, the relay (rust/kerbside-proxy/src/relay.rs) is inspection-first: it frames every message by its 6-byte MessageHeader and passes it through a Policy before forwarding. That policy is EnforcingPolicy (policy.rs): a real, enforcing application-level SPICE firewall, on by default.

Per-message pipeline

For every framed message the relay's pump runs:

1. Header parse. The 6-byte header (message_type: u16, message_size: u32) is parsed first; nothing below inspects the body until this succeeds.
2. Policy::check_header (L0, pre-body). Called exactly once per message, before its body is buffered, so an over-cap or over-rate message is refused without ever accumulating it:
3. a per-(channel, direction) size cap — tight (4 KiB) on the inputs-client and cursor-client directions (key/mouse events are small and fixed), generous (16 MiB) everywhere else (the display server bulk direction and any channel with no modeled grammar);
4. a coarse per-direction rate/throughput ceiling (fixed window), disabled by default — a deployment can opt in once real traffic patterns justify a value.

These policy caps sit below the relay's own unconditional MAX_MESSAGE_SIZE (16 MiB) absolute frame guard, which the relay enforces unconditionally regardless of policy — a peer that claims an absurd message_size is refused before the relay would buffer gigabytes waiting for a body that never arrives. 3. Policy::inspect (L1). Once the full body has been buffered, the message type is classified against a compiled-in per-channel, per-direction allowlist (allowlist.rs), derived from the ryll shakenfist-spice-protocol name tables unioned with the SPICE common-base opcodes (server SPICE_MSG_* 1..=7, client SPICE_MSGC_* 1..=6, valid on every channel). Main, display, inputs, cursor, and playback have their own tables; usbredir rides the spicevmc table, and port/webdav ride the port table (the spicevmc set plus PORT_INIT and PORT_EVENT). Record, smartcard, and tunnel have no modeled grammar — they classify as ChannelUnmodeled and get L0-only enforcement plus observe-only handling for their message types (never a type-based terminate), rather than treating every type on those channels as a violation. 4. Verdict. Forward (write the original framed bytes unchanged) or Terminate (flush and end the whole relay — a single SPICE channel is one duplex TCP connection, so either direction ending ends the session). Drop is a variant on the Verdict enum reserved for future L2/L3 use (e.g. defanging); no rule emits it today, because silently dropping a mid-stream SPICE message would desynchronise the channel.

Why the rules are what they are

• A disallowed type terminates in both directions, server→client included, so a hypervisor sending malformed messages to exploit a client is cut off. When a legitimate type is refused, the fix is to add it to ryll's tables, never to weaken the verdict.
• The allowlist is only as good as ryll's name tables. A type those tables name is allowed, so a wrong entry fails open, and a missing one terminates real sessions. In July 2026 the main and display tables lacked several opcodes and mislabelled INVAL_ALL_PIXMAPS, which would have ended ordinary remote-viewer sessions; ryll PR #141 fixed them before enforcement shipped.
• Message order within a channel is not checked. Rules such as "init before data" carry a high risk of refusing legitimate traffic, so they are left to future body (L2) validation; L1 checks only the type and whether the channel is permitted.
• The idle-read timeout (15 minutes) is a backstop against a silent client holding a connection permit forever. A client that trickles bytes slowly is bounded instead by the size of the permit pool and the 16 MiB absolute frame guard.

The rules were checked against real traffic before they shipped enforcing. On 2026-07-07 a remote-viewer session through the proxy in warn-only mode, across the main, display, inputs and cursor channels with about 1.17 MB relayed, recorded no verdicts at all and came nowhere near a size cap, which is why no cap has been tightened. The direct-qemu lane now holds every pull request to the same result with ryll and a WebDAV channel; see Validating the SPICE firewall.

Warn-only mode

FirewallPolicy.mode (EnforcementMode::Enforce default, or WarnOnly) is the single place the enforcement decision is made (EnforcingPolicy::apply). In Enforce, a rule's blocking verdict is applied and recorded action=enforced. In WarnOnly, the SAME blocking verdict is downgraded to Forward — the session is never actually blocked — but still recorded action=observed plus a tracing::warn!. This lets an operator run real traffic through a deployment and see exactly what Enforce would have tripped before switching to it; it is a mode, not a grace period, since the compiled default ships Enforce.

A forbidden channel type (a whole channel the deployment's FirewallPolicy.permitted_channels excludes) is denied earlier, in session.rs, before any relay is set up — a protocol-correct PermissionDenied to the client plus an audit event, the same shape as the existing unknown-channel-type rejection.

Audit and metrics

Firewall violations must not become one audit event per message — a hostile or broken peer could otherwise flood the audit log. Instead, both relay directions share one lock-free per-connection VerdictTally (atomic counters keyed by (rule, action)); relay::run reads it once after the two pumps finish and, if anything was recorded, emits a single coalesced summary RecordAuditEvent for the whole connection (e.g. "Firewall verdicts this connection: disallowed_type (enforced=2, observed=0)"). Every verdict is also exported live as the Prometheus counter kerbside_proxy_firewall_verdicts_total{channel,direction,rule,action}, where rule is one of disallowed_type, unmodeled_type, size_cap, or rate_cap, and action is enforced or observed.

Policy delivery

Python continues to own policy; the Rust proxy only enforces it ("Python decides policy; Rust enforces it"). The tunable knobs — mode and permitted channels — are delivered per-connection in the AuthorizeConnection gRPC reply as a FirewallPolicy protobuf message, built from Python's FIREWALL_MODE/FIREWALL_PERMITTED_CHANNELS config (see docs/configuration.md). The L1 allowlist tables themselves are not delivered over gRPC: which message types are structurally valid on a channel is a fact about the SPICE protocol, not a deployment policy, so they are compiled into the proxy. Size caps, the rate ceiling, and per-verdict severities keep their compiled defaults (no gRPC config surface yet).

Not implemented anywhere in the Rust proxy: L2 body validation (scancode ranges, clipboard/file-transfer/usbredir device-class filtering), session recording, and L3 rewriting/injection.

Process Supervision and Session Termination

The Python daemon runs the Rust proxy as a child process, and API-driven session termination drops in-flight connections rather than only blocking new ones.

Process model: the daemon supervises the Rust proxy as a child

kerbside daemon run supervises the Rust proxy binary as a child:

flowchart TD
    daemon["kerbside-daemon (main.py:daemon_run)<br/>binds the gRPC UDS server FIRST"]
    proxy["kerbside-proxy (async tokio tasks)<br/>the Rust binary (rust/kerbside-proxy/)<br/>dials the gRPC UDS at startup (ClearNodeChannels)<br/>and lazily thereafter"]

    daemon -- "subprocess.Popen" --> proxy

• The gRPC UDS server is bound before the child is launched, because the Rust proxy dials it at startup and would fail if the socket did not yet exist. (A subprocess child, unlike a fork, does not inherit the gRPC server's fds/threads.)
• kerbside/proxy_supervisor.py's find_proxy_bin() resolves the binary (env KERBSIDE_PROXY_BIN → PATH → the in-repo target/{release,debug} dev build dir) and build_proxy_argv() maps config to the proxy's CLI flags (--vdi-address, --secure-port, --cert, --metrics-address, etc.) — the firewall knobs are excluded, since those are delivered per-connection over gRPC (see the firewall section above), not on the command line.
• The daemon polls the child's liveness every second and forwards SIGTERM with a 15-second deadline before SIGKILLing it; exits non-zero if the child dies unexpectedly. The 15-second deadline is sized above the proxy's own 10-second graceful-drain window (below), so the daemon gives the proxy a chance to drain before forcing it.

How the binary gets there: packaging

find_proxy_bin()'s middle leg — shutil.which('kerbside-proxy') — is what resolves the binary in a real deployment. The crate is published to PyPI as a separate kerbside-proxy package: a maturin bindings = "bin" wheel (rust/kerbside-proxy/pyproject.toml) whose compiled binary is laid into the wheel's *.data/scripts/ directory, which pip installs onto PATH.

The committed pyproject.toml does not carry an exact pin. It carries a dev-inclusive FLOOR — kerbside-proxy>=0.4.0.dev0 — anchored on the # KERBSIDE_PROXY_PIN marker. Naming a .dev0 version in the specifier is what opts pip into pre-release resolution: without it, pip would never consider a dev wheel a valid match for a plain >= floor. This means a pip install from a git checkout of kerbside (not a tagged release) resolves whatever kerbside-proxy is newest on PyPI at install time — a tagged release, or a rolling dev wheel.

Unreleased kerbside still needs a working proxy to test against, which is what .github/workflows/dev-proxy-wheel.yml provides: it builds and publishes a PEP 440 dev wheel of kerbside-proxy (e.g. 0.4.1.dev163, versioned by setuptools_scm) to PyPI, unattended, on pushes to develop that are path-filtered to the binary's inputs (rust/**, kerbside/rpc/kerbside.proto, tools/build-proxy-wheel.sh, tools/stamp-dev-proxy-version.sh, tools/gen-protos.sh, and the workflow file itself) — minus rust/kerbside-proxy/Cargo.lock, since a lockfile-only bump cannot change the binary's interface or contract hash. Dev wheels carry build provenance attestations, the same as tagged releases, but they are not Sigstore tag-signed — there is no v* tag for a dev build to sign. See "Configure Dev Release Publishing" and "Dev releases" in RELEASE-SETUP.md. These dev releases accumulate on PyPI over time and are pruned manually; see "Pruning dev releases" in RELEASE-SETUP.md.

At release time the two packages release in lockstep from a single v* tag: setuptools_scm gives kerbside its version, and tools/stamp-proxy-version.sh stamps that same version into the crate (for maturin) and REPLACES the committed floor line in pyproject.toml with the exact kerbside-proxy==<version> pin. A stamped release tree is exact-pinned; the develop branch is not. tools/build-proxy-wheel.sh builds prebuilt manylinux_2_28 wheels for x86_64 and aarch64 (the latter cross-compiled with maturin --zig, so no aarch64 build host is required); no source distribution is published, so an unsupported platform gets a clean pip error rather than a doomed source build. In development you bypass all of this: find_proxy_bin() falls through to the in-repo cargo build output, or you set KERBSIDE_PROXY_BIN explicitly. See RELEASE-SETUP.md.

The gRPC contract handshake

Because the committed tree is not exact-pinned, an installed kerbside-proxy binary is not guaranteed to speak the same gRPC contract as the kerbside it is paired with — the pin only guarantees that at release time. The handshake is the backstop for every other case (dev installs, a stale local build, an operator mismatching versions by hand):

• tools/gen-protos.sh (run via tox -egenprotos, whenever kerbside/rpc/kerbside.proto changes) writes the sha256 of the proto's raw bytes to a generated constant, CONTRACT_HASH, in the committed kerbside/rpc/contract.py.
• rust/kerbside-proxy/build.rs computes the same sha256 from its copy of kerbside.proto at compile time and embeds it in the binary; the binary prints it and exits when run with --contract-hash.
• Before launching the proxy, launch_rust_proxy() (kerbside/proxy_supervisor.py) calls check_contract(), which runs the resolved binary with --contract-hash and compares the result to CONTRACT_HASH. A mismatch — including a binary that predates the --contract-hash flag entirely, which reports as "unknown" rather than a hash — raises RuntimeError naming both hashes and four remediations (upgrade the wheel, rebuild the local Rust tree, point KERBSIDE_PROXY_BIN at a matching binary, or bypass the check).
• KERBSIDE_SKIP_CONTRACT_CHECK is that bypass: set it to a recognised truthy value (1, true, yes, or on, case-insensitively) and the mismatch is logged as a warning instead of refused. It exists for debugging only and is not a supported deployment posture.

The handshake is a compatibility check, not an integrity control. It detects accidental build skew — a stale local cargo build, a wheel that did not upgrade with the package, a hand-mismatched pair — and it detects nothing at all about a substituted or tampered binary. An attacker who can replace the binary controls what it prints when asked for its hash, and more fundamentally the check has to execute the untrusted binary in order to decide whether to trust it, so anyone in a position to defeat it already has code execution. Nothing downstream of this may treat a passing handshake as evidence that the binary is the one this package shipped with.

Session termination: dropping in-flight connections

Removing or expiring the DB token blocks a new connection attempt, but it does nothing to a client that is already connected: that client keeps its channels open until it disconnects itself. Dropping a live session needs a path from "the API terminated this session" to "the proxy holding that session's sockets finds out", so ConsolesTerminate/SessionTerminate do both.

The distributed-deployment constraint. Kerbside can run with the REST API and the proxy processes on different machines (the proxy is often sized separately from the API), and proxies can sit behind a load balancer so that different channels of the same session land on different proxy nodes. The only component every one of these can reach is the shared MariaDB — there is no API→proxy or proxy→proxy RPC across machines. The ProxyControl gRPC stream, by contrast, is strictly local: one stream between a daemon and the single proxy it supervises on the same host, over the same UDS as every other RPC in the table above. Termination therefore has to be a DB-mediated intent that each node acts on independently for the channels it happens to hold. Two simpler designs were rejected: inferring termination from the token's absence would conflate it with natural expiry, which must never drop a live session, and an RPC from the API to a daemon cannot reach a proxy on another machine or behind a load balancer.

flowchart TD
    api["REST API<br/>(may be elsewhere)"]
    db[("MariaDB<br/>the only shared bus")]

    api -- "INSERT session_terminations(S)" --> db

    %% Node B is declared FIRST on purpose: as of 2026-08, the mermaid
    %% version GitHub renders lays subgraphs out in reverse declaration
    %% order, so this is what puts node A on the left. That is observed
    %% behaviour, not documented -- verify after a mermaid bump. Left
    %% versus right carries no meaning either way.
    subgraph nodeB["Proxy node B — holds other channels of session S"]
        direction TB
        daemonB["Daemon stream:<br/>ProxyControl TerminateSession(S)"]
        registryB["Rust proxy:<br/>SessionRegistry.terminate(S)"]
        relayB["relay::run's select! sees token.cancelled()<br/>→ teardown, once per channel this node holds"]
        daemonB --> registryB --> relayB
    end

    subgraph nodeA["Proxy node A — holds channels 1, 2 of session S"]
        direction TB
        daemonA["Daemon stream:<br/>ProxyControl TerminateSession(S)"]
        registryA["Rust proxy:<br/>SessionRegistry.terminate(S)"]
        relayA["relay::run's select! sees token.cancelled()<br/>→ teardown, once per channel this node<br/>holds — here, 2 relays end"]
        daemonA --> registryA --> relayA
    end

    db -- "polls get_terminations_for_node(A)" --> daemonA
    db -- "polls get_terminations_for_node(B)" --> daemonB

Concretely:

1. API (api.py, ConsolesTerminate/SessionTerminate): expires or removes the token, and calls db.request_session_termination(session_id, reason), which upserts a row into the session_terminations table (session_id primary key, requested_at, optional reason; migration c4e7a1b9d2f3). Idempotent — re-terminating an already-terminated session just refreshes requested_at.
2. Daemon (servicer.py::ProxyControl, one stream per local proxy): each poll interval, calls db.get_terminations_for_node(NODE_NAME) — an intersection of session_terminations with this node's live proxychannels rows — and yields a TerminateSession(session_id) for every id not already sent on this stream, interleaved with Heartbeats so the stream stays alive between events. A DB error is logged and the loop continues rather than killing the stream. Natural token expiry (no explicit termination) is never pushed — expiry only blocks new connections.
3. Rust proxy (session.rs::SessionRegistry, rpc.rs:: run_proxy_control): a Mutex<HashMap<session_id, CancellationToken>> refcounted across the session's live channels (one entry per session, incremented on each channel's registration, decremented and removed on the last channel's teardown). Receiving TerminateSession cancels the token; relay.rs's select! gained a third arm on token.cancelled() alongside the two pumps, so every channel of the session this node holds tears down cleanly (no error, same shape as a protocol-level Terminate verdict) and the client is disconnected. Cancelling an unknown or already-torn-down session is a no-op, so a late or duplicate push (another node's poll cycle, a retry) is safe.
4. Reaper (daemon maintenance loop, reap_session_terminations): deletes session_terminations rows older than 5 minutes — long past the point every node has polled and acted — keeping the table small.

A merely-expired (not explicitly terminated) token is never pushed as a TerminateSession, so natural token expiry does not tear a live session down mid-use — only an explicit API terminate does.

Graceful drain on shutdown

On SIGTERM, the proxy's shutdown_signal future resolving triggers the same teardown path used for termination: it stops the listeners from accepting new connections, then calls SessionRegistry::terminate_all() (cancelling every live session's token at once) and waits up to a 10-second deadline for the active-connection count (the same gauge behind /metrics) to reach zero before returning. The 10-second drain deadline is deliberately shorter than the daemon's 15-second SIGTERM-to-SIGKILL window above it, so a supervised restart lets in-flight sessions finish tearing down on their own terms rather than being cut off mid-drain by the daemon's SIGKILL.

Verifying it live

The mock harness (direct-qemu-harness.md) drives this end to end against a real client: the mock gRPC server's ProxyControl stream emits a one-shot TerminateSession a configurable number of seconds after the first authorization (MOCK_GRPC_TERMINATE_AFTER), standing in for the API→DB→daemon-poll leg so the harness can exercise just the proxy side live. A recorded run against remote-viewer shows all four of its channels (main, display, inputs, cursor) torn down by a single TerminateSession event.

Related Documentation

• Protocol Overview - SPICE protocol introduction
• Link Protocol - Connection handshake details
• Channel Protocols - Per-channel message formats

📝 Report an issue with this page