Diagnostics and instrumentation

Ryll carries a set of facilities for understanding what a session did: statistics collection, an in-memory traffic ring buffer, channel state snapshots, bug report assembly, a live traffic viewer, and a capture mode that records per-channel protocol pcaps and H.264 video of the display to a directory.

Statistics and Instrumentation

Ryll tracks:

• FPS: Sliding-window frames-per-second derived from DisplayMark boundaries (true frame completions), not individual draw operations. The window keeps the most recent 120 timestamps for an accurate short-term reading.
• Bytes in/out: Network throughput per channel
• PING interval: Client-observed interval between server PINGs on the main channel, in milliseconds, shown in the status bar under that label. SPICE has no client-originated probe (SPICE_MSG_PING is server→client only), so ryll cannot measure absolute network RTT. Instead, the main-channel PING handler records Instant::now() and emits the gap to the previous PING as a sample. The number includes the server's send cadence and the client's receive turnaround; spikes indicate a network or server stall. Sparkline mirrors the bandwidth one (60-sample rolling history, amber bars). Implemented via LatencyTracker in ryll/src/app/stats.rs.
• Bandwidth sparkline: A rolling 60-sample history of bytes/sec is displayed in the status bar as a small bar chart. Channel read loops increment a shared AtomicU64 byte counter; the BandwidthTracker in ryll/src/app/stats.rs samples it once per second and renders the sparkline.
• Runtime metrics in bug reports: each bug-report ZIP includes a runtime-metrics.json file with process and per-thread CPU%, RSS, and VmSize sampled over a 2-second window. On Linux the data comes from /proc/self/stat, /proc/self/status, and /proc/self/task/*/. On macOS process-level data comes from task_info(MACH_TASK_BASIC_INFO) and per-thread data from task_threads + per-port thread_info(THREAD_BASIC_INFO) + thread_info(THREAD_IDENTIFIER_INFO) (for the stable 64-bit thread id) + pthread_getname_np (for the name). The Mach port array is held in a MachThreadList RAII guard so port references are released on every exit path. Other platforms emit a graceful "unavailable" payload. The MacOS enum variant has the same JSON shape as Linux, distinguished by the platform field. Implemented in shakenfist-spice-renderer/src/metrics.rs.

This instrumentation is the primary purpose of ryll -- measuring kerbside proxy performance.

Traffic Ring Buffer

Every SPICE message (sent and received) is recorded in a per-channel ring buffer regardless of whether --capture is active. The ring buffer retains the most recent traffic up to a 50 MB total cap, allocated by weight per channel rather than split evenly:

Channel	Cap	Weight	Coverage at measured desktop rates
display	32 MB	16	~16 s @ 2 MB/s typical, ~5 s @ 6 MB/s peak
usbredir	4 MB	2	session-long when idle (active transfers exceed any cap)
playback	4 MB	2	many minutes of audio
cursor	4 MB	2	many minutes
main	4 MB	2	hours
inputs	2 MB	1	hours

Weighting display most heavily is load-bearing for the notification-snapshot feature: a snapshot captured at notification-fire time covers the run-up to the event for users to file in a bug report.

Each entry stores structured metadata (channel name, direction, message type ID and human-readable name, wire and payload sizes, timestamp) alongside the pcap frame bytes for export. SPICE messages that exceed the IPv4 single-frame limit (~64 KB) are split into multiple TCP segments via the shared capture::segment_payload helper; each TrafficEntry carries its first segment in pcap_frame and any trailing segments in additional_segments: Vec<Arc<[u8]>>. The Arc<[u8]> choice for both fields makes ring-buffer entry clones O(N atomic refcount bumps) rather than O(total bytes), which is what enables the notification-snapshot store's cheap deep-copy at fire time.

The TrafficBuffers struct in ryll/src/bugreport.rs holds all six per-channel TrafficRingBuffer instances behind Mutex<> and is shared via Arc<TrafficBuffers> between all channel handler tasks and the UI thread. This supports both bug-report export, the live traffic viewer, and the snapshot-on-notification path. (The webdav channel is intentionally absent — its handler does not call traffic.record_* today; tracked as a follow-up in the dogfooding feedback plan.)

Channel State Snapshots

Each channel handler maintains an Arc<Mutex<T>> snapshot struct that captures the channel's mutable state. The snapshots are updated in-place after every batch of processed messages and after every sent message. All snapshot structs derive serde::Serialize so they can be written to JSON for bug reports.

Snapshot struct	Channel	Key fields
DisplaySnapshot	Display	Image cache size and 64 most-recent IDs, per-opcode recv/send maps, unknown-opcode counter, stream lifecycle counters (created/destroyed/rejected), recent decode results (last 20) with per-decode wall-time, decode/socket/ACK diagnostic counters, pcap writer-queue drop counter, ACK state, bytes in/out, image cache statistics (cap in bytes, LRU eviction count, bytes evicted) from BoundedImageCache
InputsSnapshot	Inputs	Button state, motion count, recent input events (last 50), per-opcode recv/send maps, unknown-opcode counter, pcap writer-queue drop counter, bytes in/out
CursorSnapshot	Cursor	Cursor cache contents, ACK state, per-opcode recv/send maps, unknown-opcode counter, pcap writer-queue drop counter, bytes in/out
MainSnapshot	Main	Session ID, mm_time, keepalive, per-opcode recv/send maps, unknown-opcode counter, pcap writer-queue drop counter, bytes in/out
PlaybackSnapshot	Playback	Per-session audio metadata (PlaybackSessionInfo), start/stop counts, data-packet and decode counters, unsupported-codec drop counter, PCM byte counts, recent decode-duration ring (cap 64), device-side atomics (callbacks, underruns, platform xruns, ring overflows, samples consumed), output device in use (PlaybackOutputInfo), last output failure and failure/rebuild/reroute counters, server volume/mute/latency params, per-opcode recv/send maps
UsbredirSnapshot	Usbredir	Redirected device list (RedirectedDevice), device connect/disconnect totals with timestamps, server/client capability bitmasks from hello handshake, per-opcode recv/send maps, unknown-opcode counter, bytes in/out
WebdavSnapshot	Webdav	Transport common + per-opcode recv/send maps (pending)
AppSnapshot	App (UI)	FPS, bandwidth, surfaces, cursor position, uptime, video encoder-queue drop counter, render-side mpsc-queue lag aggregates for ImageReady* and DisplayMark events, ryll's own volume and mute (null where no GUI maintains the snapshot)

All channel snapshots share an eight-field transport common baseline (bytes_in, bytes_out, last_recv_ts_secs, last_send_ts_secs, ping_recv_count, pong_send_count, last_ping_recv_ts_secs, writer_dropped_count) plus four baseline additions (messages_recv_by_opcode, messages_send_by_opcode, last_unknown_opcode, unknown_opcode_count), produced by the shared OpcodeCounters type. Only opcodes the protocol crate has a name for get their own map entry — a server-chosen opcode this build does not recognise is folded into unknown_opcode_count so the map cannot be grown from the wire. See the channel diagnostics audit for the full audit matrix and minimum-baseline rationale.

MainSnapshot also records four fields for diagnosing a stalled UI: events_dropped_count, events_dropped_by_kind and last_event_drop_ts_secs count the ChannelEvents the main channel discarded because the renderer's event queue stayed full for longer than MAIN_EVENT_SEND_TIMEOUT (per ChannelEvent variant name, with the session-relative time of the latest drop); a non-zero count means the UI stopped draining events. Mouse mode and vdagent state are not events, so they are never counted here: the main channel publishes them through the renderer's SessionState watch channels, which hold the latest value and cannot drop it. server_mouse_mode is the mouse mode the server last announced in MAIN_INIT or MOUSE_MODE (1 = server/relative, 2 = client/absolute); compare it with mouse_mode in the app snapshot, as a mismatch means the client is sending the wrong kind of pointer message.

The ChannelSnapshots struct in ryll/src/bugreport.rs holds the four channel snapshot Arc<Mutex<T>> values and is created alongside TrafficBuffers in run_connection(). The AppSnapshot is maintained separately by the RyllApp event loop.

Updates hold the mutex only briefly (copying a handful of scalars and small collections), so contention with the UI thread is negligible.

Bug Report Assembly

BugReport in ryll/src/bugreport.rs assembles a self-contained zip file from the ring buffer, channel snapshots, and app state. The zip contains:

ryll-bugreport-YYYY-MM-DDTHH-MM-SSZ.zip
├── metadata.json         # report type, description, ryll version,
│                         #   platform, target, timestamp
│                         #   (submit), triggered_at (dialog-open),
│                         #   session_uptime_secs (submit),
│                         #   triggered_uptime_secs (dialog-open)
├── session.json          # AppSnapshot (FPS, bandwidth, surfaces,
│                         #   held keys and recent key events)
├── channel-state.json    # snapshot of the affected channel
├── traffic.pcap          # ring buffer pcap (capture feature only)
├── screenshot.png        # trigger-time full surface (Display only)
├── screenshot-region.png # submit-time crop at the selected region
│                         #   (Display only, when a region was drawn)
└── runtime-metrics.json  # process and per-thread CPU%, RSS, VmSize
                          #   sampled over a 2-second window at
                          #   report-creation time (Linux only;
                          #   non-Linux platforms record
                          #   available:false with a reason)

Report types are Display, Input, Cursor, Connection, Usb, Audio, and Pedantic, each mapping to one SPICE channel or the --pedantic observer path. Connection covers the main channel only; an audio problem needs Audio, whose channel-state.json is the playback snapshot. BugReport::new() samples runtime metrics over a 2-second window (blocking the caller), then gathers and serialises all data synchronously. BugReport::write_zip() writes the zip to the capture directory's bug-reports/ subdirectory (if --capture is active) or the current working directory.

No caller may run that sample on a thread that has to stay responsive. The auto-snapshot loop wraps the whole BugReport::new() call in tokio::task::spawn_blocking; the --pedantic observer samples in spawn_blocking and hands the result to BugReport::write_pedantic().

RyllApp::finish_bug_report() is the GUI entry point. It captures the submit-time surface pixels, the trigger timestamps, and the precomputed PNG, starts the metrics sample on a dedicated std::thread, and parks the submission. RyllApp::poll_pending_bug_report() picks it up on the frame the sample lands, calls BugReport::assemble() with the sampled metrics, and writes the zip. The window keeps painting throughout, and a notification announces the start of the capture as well as its result. A second submission is refused while one is in flight.

Display bug reports carry two PNGs. screenshot.png is the surface captured the moment the dialog opens — a background std::thread PNG-encodes the cloned RGBA while the user types a description. screenshot-region.png (when a region was drawn) is a crop of the submit-time surface at the selected rectangle, encoded on the UI thread after the user finishes the drag. The two images are deliberately different moments in time.

Non-Display submissions drop the precomputed PNG even if one was captured — the dialog captures unconditionally on open (so an artefact doesn't fade while the user decides what to submit), but only includes the PNG when the user actually submits as Display.

Bug Report Dialog

Pressing F12 or using Menu → Report opens a centred modal dialog for generating bug reports. The dialog contains:

1. A privacy warning about sensitive data in reports.
2. Radio buttons to select the report type (Display, Input, Cursor, Connection, USB, Audio).
3. An optional description text field.
4. Capture and Cancel buttons.

While the dialog is open, keyboard and mouse input is not forwarded to the SPICE server. F12 is always consumed by ryll (never sent to the guest). Escape closes the dialog.

The dialog uses a two-pass pattern to avoid egui borrow checker conflicts: the UI is rendered in a closure that collects the user's action into a local variable, then the action is executed on self after the closure returns.

After a successful report, a transient status message ("Bug report saved to ...") is displayed in the status bar for 5 seconds.

Display region selection

When the user selects "Display" and clicks "Capture", the dialog closes and the app enters region selection mode:

1. A translucent instruction banner appears at the top of the surface: "Click and drag to select the affected region. Press Escape to skip."
2. The OS cursor changes to a crosshair (the SPICE cursor overlay is hidden).
3. The user drags a rectangle; a translucent red overlay shows the selection.
4. On mouse release, the report is generated with the region coordinates in the metadata. A click without a drag stays in region selection and says so in a notification.
5. Pressing Escape skips selection and generates without a region.

The dialog says, while Display is selected, that Capture will ask for a region, and entering region selection also posts a notification, because a user who misses the banner otherwise sees a window that ignores the guest and appears stuck.

Every way out of region selection writes the report, and the log records which one it was (app: region selection completed, skipped with Escape, abandoned by a reconnect, abandoned by a lost connection or abandoned by the window closing), next to app: bug report waiting for region selection on the way in. A reconnect or a lost connection clears the surface being selected from, and closing the window ends the session, so each writes the report without a region rather than dropping it. Closing the window also writes a report still waiting for its runtime metrics sample, after at most a second past the end of its sample window.

Only one report is captured at a time. Pressing Capture on a Display report while an earlier report is still sampling its metrics (the two seconds after its Capture) does not enter region selection: the report is refused at once with "A bug report is already being captured", so a selection never ends with a report that cannot be written.

While a selection is in progress, Menu → Report and Menu → Screenshot are disabled, as F12 and F8 are, and the USB and connection error dialogs' report buttons are refused with a notification: opening a second report would replace the pending one's trigger snapshot.

Trigger-time snapshot

On dialog open, RyllApp::begin_trigger_snapshot clones the largest surface's RGBA and spawns a named std::thread (ryll-bugreport-png) that PNG-encodes into a shared Arc<Mutex<Option<Result<Vec<u8>>>>>. The submit path (finish_bug_report → take_trigger_for_submit) consumes the encoded bytes via try_lock, falling back to a live encode if the encoder hasn't finished. Close-without-submit paths (Escape, Cancel, F12 toggle-off) drop the Arc; the thread finishes into what becomes garbage.

Keyboard and mouse input is not forwarded to the SPICE server during selection. Coordinates are clamped to the surface bounds.

Live Traffic Viewer

Pressing F11 or using Menu → Traffic toggles a right-side panel showing a live feed of recent SPICE protocol messages from the ring buffer.

The viewer collects entries from all four channels via TrafficBuffers::recent_view_entries(), which returns lightweight TrafficViewEntry structs (no pcap frame data). Entries are cached in RyllApp and refreshed every 250ms to minimise mutex contention.

Features: - Channel filters: checkboxes to hide/show individual channels - Pause/Resume: freezes the display for inspection - Auto-scroll: sticks to the bottom when not paused - Colour-coded channels: main=blue, display=green, inputs=orange, cursor=purple

Each row shows: relative timestamp, channel name, direction arrow (sent/received), message name, and wire size.

F11 is consumed by ryll and not forwarded to the SPICE server.

Capture Mode

When --capture <DIR> is specified, ryll records:

Session metadata

metadata.json is written at session start with platform details (OS, architecture), ryll version, and a single target field (from ConnectionConfig::display_target(), so a tunnelled connection's proxy address is shown with the target redacted rather than the raw host:port). For a direct connection target is host:port with the port actually dialled, which is the TLS port when the .vv has one; target_tls_port records that TLS port on its own, and is null for a plain connection. Bug-report zips carry the same target field in their metadata.json. This makes capture directories self-describing when shared for bug reports or debugging.

Protocol capture (pcap)

Each SPICE channel writes a separate pcap file (main.pcap, display.pcap, cursor.pcap, inputs.pcap, usbredir.pcap, webdav.pcap) containing decrypted SPICE mini-header messages wrapped in fake TCP/IP headers. Wireshark can open these directly.

Implementation: capture::PcapChannelWriter per channel, using pcap-file for pcap output and etherparse for header construction. Packets are recorded in send() and the read loop of each channel handler. Writers use unbuffered I/O (no BufWriter) so every packet hits disk immediately.

Clipboard text crosses the main channel in AGENT_DATA messages, so whatever is copied between host and guest is in main.pcap, and in the main channel's ring buffer that bug reports include. The GUI has always synced the clipboard. Headless and web sessions do so only when they run a control socket (--control-socket), whose clipboard verbs need it; such a session fetches every guest copy at once, so guest clipboard text can appear in their captures and bug reports too, as can text set with clipboard_set. Without a control socket they never request the guest's clipboard.

Large SPICE messages (e.g. uncompressed display updates) can exceed the IPv4 maximum packet size (65535 bytes). The pcap writer splits these into multiple TCP segments with sequential sequence numbers, so Wireshark can reassemble them and the pcap file never triggers a length-overflow panic.

Display capture (video)

display.mp4 contains an H.264 encoded video of the primary surface (surface 0). Frames are emitted on MARK boundaries with real timestamps for variable-rate playback. Only the GUI records it: frames come from the egui frame loop, and headless mode, which paints no surface, writes the pcaps alone.

Implementation: capture::VideoWriter is owned exclusively by a dedicated OS thread (ryll-capture-video, running video_writer_loop) started in CaptureSession::new. The egui call site (CaptureSession::frame) is a non-blocking try_send into a bounded channel (cap VIDEO_QUEUE_CAPACITY = 8) that returns bool; false signals the encoder queue was full and the frame was dropped. The drop counter lands in AppSnapshot::video_drop_count. The encoder thread lazy-inits VideoWriter from the first surface-0 frame, encodes RGBA → YUV420 → H.264 via openh264, and muxes into MP4 via the mp4 crate. On sender drop the thread drains, calls VideoWriter::close() to write the MP4 moov atom, then exits. Pcap follows the same pattern on ryll-capture-pcap (pcap_writer_loop) with cap PCAP_QUEUE_CAPACITY = 1024.

The writers are threads rather than tokio tasks on purpose. main() creates the CaptureSession before any tokio runtime exists, and in GUI mode one session outlives every per-connection runtime (each reconnect builds a new one), so the writers cannot belong to a runtime. Their work is blocking file I/O and encoding in any case.

The capture session is Arc<CaptureSession> shared across all channels and the app. When --capture is not specified, the field is None and all capture code paths are skipped. The CaptureSession uses an AtomicBool guard to ensure close() is idempotent -- it may be called both explicitly during shutdown and again from the Drop implementation. close() does not block: it drops both writer senders and returns, and the writer threads finish on their own (pcap flush is a no-op since pcap I/O is unbuffered; MP4 writes the moov atom). close_and_wait() also waits for both threads to exit; the exit paths in main call it so the MP4 is finalised before the process exits, even while a detached GUI connection thread still holds a clone of the session. The wait gives up with a warning after five seconds, so a wedged encoder or a stalled disk leaves an unfinalised MP4 rather than a process that will not exit. The egui update path only ever calls the non-blocking close(). Separately, Drop waits the same way, so the MP4 is also finalised once the last Arc<CaptureSession> is gone. MP4 finalisation is still not synchronous with close() itself — a bug report assembled within milliseconds of close may see an unfinalised MP4. The troubleshooting guide records the trade-off rationale.

Auto-snapshot on channel disconnect

Because oVirt and Kerbside hand out one-time-use SPICE tickets, any channel that disconnects mid-session is permanently lost — the client cannot reconnect that channel without a fresh ticket. To make those events diagnosable after the fact, every ChannelEvent::Error or ChannelEvent::Disconnected triggers a best-effort auto-snapshot via BugReport::write_disconnect. The zip carries a disconnect-cause.json record alongside the usual pcap / channel-state / metadata files. The record names the channel that fired, captures whether the main-channel client-side keepalive timeout fired (distinguishing "we timed ourselves out" from a server RST), and embeds a per-channel diagnostics map so a maintainer can compare the dropped channel against the others' last-known traffic state. ryll sends no application-level keepalive on any channel: an idle channel is kept alive by TCP keepalive on the socket and by answering the server's PINGs.

A 60 s cooldown bounds disk usage during a disconnect storm; the cooldown is updated even on write failure so a misconfigured output directory does not retry on every disconnect event. Snapshots land in (in order of preference) --bug-report-dir, <--capture>/bug-reports/, or the current working directory. The same resolution chain is used for the manual F12 / Menu → Report path. Runtime metrics are recorded as unavailable in this path — sampling them on the GUI thread would freeze the UI for ~1 s, and the pcap and channel snapshots are the load-bearing diagnostic data anyway.

Notification snapshots and the "file from notification" button

Every notification entry in the side panel carries a per-row "File…" button. Clicking always produces a bug-report zip; the variant depends on whether a live snapshot of the traffic-buffer state exists for that notification.

On every RyllApp::push_notification call, after the NotificationStore::push returns the entry id, the app captures a cheap deep-copy of TrafficBuffers keyed by id. The clone is O(N atomic refcount bumps) thanks to the Arc<[u8]> for pcap_frame and the Vec<Arc<[u8]>> for additional_segments. The store is bounded: at most 5 active snapshots, 60 s TTL. Oldest is evicted on overflow; expired entries are pruned both at capture time and once per second from the GUI tick.

When the user clicks the button:

• If notification_snapshots.take(id) returns a snapshot, the report uses it and metadata.json records report_type.snapshot_state: "AtFire". The zip filename includes atfire. The pcap reflects the ring contents from the moment the notification fired.
• If the snapshot is missing (expired or evicted), the report uses the current TrafficBuffers and tags PostEventOnly / postevent. Useful, but without the run-up to the event.

Auto-snapshot mode (--auto-snapshot-interval)

A flight-data-recorder mode. When --auto-snapshot-interval N is set, a background tokio task fires a complete bug report every N seconds into <bug-report-dir>/auto-snapshots/. This captures full session state regardless of whether the operator notices a symptom in real time — useful for intermittent issues like audio silences that last only 30 seconds.

Key design points:

• BugReportType::AutoSnapshot — a new variant whose channel_name() returns "all". This causes BugReport:: assemble to embed a merged JSON object in channel-state.json containing every channel's snapshot (display, inputs, cursor, main, playback, usbredir, webdav). The traffic pcap also covers all channels via TrafficBuffers::drain_all_pcap_bytes().
• AutoSnapshotState in ryll/src/auto_snapshot.rs bundles the Arc handles the task needs (traffic, channel snapshots, app snapshot, notifications, target, output dir, cap, byte cap, interval). All are already Arc-backed on RyllApp.
• The task runs in its own std::thread with a dedicated tokio::runtime::Builder::new_current_thread runtime, spawned on ChannelEvent::SessionInitialized. reconnect() replaces the traffic and channel-snapshot Arcs wholesale, so each session gets its own task: the app raises the outgoing task's cancel flag at disconnect and again before spawning the replacement, so at most one auto-snapshot task is running at a time. Nothing joins those threads. Every retire runs on the egui UI thread, and a thread already inside a zip write does not observe the cancel flag until it finishes, so a join would freeze the GUI for seconds. Outgoing handles are held in a retiring set instead, reaped non-blockingly with JoinHandle::is_finished, and a session declines to spawn at all once more than four are still winding down (logged at warn!). If the dedicated runtime cannot be built, the mode simply does not start for that session and logs a warning.
• Two independent bounds keep a hostile server from driving respawns. A repeated SPICE_MSG_MAIN_INIT naming the session already in hand is discarded as a duplicate. Because the session id is server-chosen, that check alone is defeated by alternating ids, so a second guard ignores any SessionInitialized arriving less than five seconds after the last accepted one while still connected. A genuine re-link without a visible disconnect is rare enough to tolerate the delay; the cost to a hostile peer is one respawn per window rather than one per 36-byte message.
• A startup NotifySeverity::Info notification confirms the mode is active: "Auto-snapshot mode enabled — every {N}s, max {cap} snapshots / {byte cap} MiB, saving to {path}".
• The stats panel renders "Auto-snapshot: {saved}/{cap}" when the mode is active; the line is hidden when disabled.
• Rolling cap: after each successful write, prune_to_cap lists ryll-auto-snapshot-*.zip in the output dir, sorts lexicographically (= chronologically by filename construction), and deletes the oldest until both budgets are satisfied — at most --auto-snapshot-cap files and at most DEFAULT_AUTO_SNAPSHOT_BYTE_CAP (512 MiB) of them. The byte budget exists because the count alone does not bound disk use: traffic.pcap is stored uncompressed and merges all six rings, so a busy session's zip approaches the 50 MiB ring budget and 20 of those are roughly a gigabyte. auto_snapshots_pruned in AppSnapshot tracks the total deleted.
• Write failures: warn! always; a NotifySeverity::Warn notification is pushed at most once per 5-minute cool-down so a persistent disk error does not spam the panel. The interval task continues regardless.
• Filename scheme: ryll-auto-snapshot-2026-05-18T20-37-42Z-T+47.3s.zip — UTC ISO timestamp (colons replaced with hyphens for Windows filesystem portability) plus session uptime suffix.

The button's visual state reflects which path a click would take: weak/dimmed text + a "snapshot expired" hover tooltip when no live snapshot exists. BugReport::write_notification mirrors write_disconnect's zip shape — same set of files, different filename prefix and metadata.

📝 Report an issue with this page