Testing¶

How Kerbside is tested: running the suite locally, the CI lanes, the Ryll-based harnesses, the oVirt console probe, the Tempest plugin, and the load-test container images.

Running the test suite¶

tox -e py3      # unit tests
tox -e flake8   # style checks on changes since HEAD~
tox -e cover    # coverage report into cover/
tox -e bindep   # OS dependency check

py3, flake8 and cover are the default envlist, so a bare tox runs all three.

Test locations:

Unit tests: kerbside/tests/unit/
Functional tests: kerbside/tests/functional/
Tempest plugin: tempest-plugin/kerbside_tempest_plugin/ (a separate releasable, driven via tools/run-tempest-tests and the openstack_matrix job in .github/workflows/functional-tests.yml)
tests/api/test_spice_via_kerbside.py — OpenStack lane only; requires a live cloud.
tests/scenario/test_sextant_scenario.py — direct-qemu lane; see Sextant scenario test.

CI tiers¶

Develop is protected by a GitHub merge queue and the lanes are split into two tiers.

The smoke tier runs on every pull request push: unit tests and lint, the direct-qemu lane, and the Shaken Fist end-to-end lane. Between them these deploy the PR's own code and relay real SPICE traffic through the real proxy, so a PR still gets end-to-end signal in tens of minutes rather than hours.
The nightly tier runs once a day against develop, and on workflow_dispatch: the oVirt and OpenStack cloud matrices, each of which builds an entire cloud from scratch. These are the expensive lanes, and their failure modes are dominated by upstream environment churn rather than by Kerbside regressions, so running them per PR bought little for what it cost. The Proxmox VE lane (proxmox-functional.yml) is in this tier too, but it also runs on pull requests that touch the Proxmox source, as an advisory check; see the Proxmox lane.

The trade is deliberate, and it has been made twice. The cloud lanes first moved from pull requests to the merge queue. They then moved out of the queue as well: in the month before, they ejected about one queue entry in ten, never once for a Kerbside regression, and kerbside-patches also dispatched the OpenStack lane on every push to its develop, re-testing code its own merge queue had just deployed with Kolla. That came to some 200 OpenStack runs a month and about 80 oVirt runs, each holding a runner and a nested cloud on sfcbr for most of an hour. A cloud-specific breakage now surfaces the morning after it merges, as a Nightly oVirt lane failing, Nightly OpenStack lane failing or Nightly proxmox lane failing issue filed by the run, rather than on the PR or in the queue. Nothing else exercises those sources end to end, so treat those issues as release blockers (see "Releasing" in development.md), and dispatch the lane on a pull request that changes a source driver.

Both tiers also skip entirely when a change touches only files no lane exercises: the review-tracking state (REVIEWS.md and the .vscode weaudit files) and docs/. Each gating workflow carries a check_paths filter job whose output the heavy jobs skip on — a filter job rather than trigger-level paths-ignore, because a required check that never reports blocks the merge forever while a skipped one satisfies it (see the comments in functional-tests.yml). The filter list is duplicated in the three check_paths jobs and in codeql-analysis.yml's trigger-level paths-ignore (safe there because CodeQL is not a required check); keep the copies in sync. One difference is deliberate: only codeql-analysis.yml skips .vscode/review-scope.toml. The scope config decides which files are reviewed at all, so a pull request narrowing it needs the same lint and automated review as a code change, and both sit behind the check_paths filter. CodeQL does not read the file, so it alone may skip it. In the merge queue only functional-tests.yml runs its filter, so a review-marks-only or docs-only queue entry skips sanity_checks too — confirmed live on the queue entry for PR 304. The one job exempt from all of this is the credential scan: a credential pasted into a code sample is still a credential, so it runs on every change.

Workflow	Runs on	Tier
`functional-tests.yml` (`sanity_checks`)	pull_request, merge_group	smoke
`functional-tests.yml` (`credential_scan`)	pull_request, merge_group, never path-filtered	smoke
`functional-tests.yml` (`docs_checks`)	pull_request, merge_group, never path-filtered	smoke
`functional-tests.yml` (`ovirt_matrix`, `openstack_matrix`)	nightly (04:30 UTC), workflow_dispatch	nightly
`proxmox-functional.yml`	nightly (05:30 UTC), workflow_dispatch, and pull_request path-filtered to the Proxmox path; see "The Proxmox end-to-end lane" below (advisory, never in the merge queue)	nightly
`direct-qemu-functional.yml`	pull_request, merge_group, nightly	smoke
`sf-e2e-functional.yml`	pull_request, merge_group, nightly (02:30 UTC), workflow_dispatch	smoke
`rust.yml`	push and pull_request, path-filtered to `rust/**` and the proto	neither (advisory)
`demo-compose.yml`	push and pull_request, path-filtered to `demo/`, `tools/demo/`, the migrations, the proto, `pyproject.toml` and `docs/installation.md`	neither (advisory)
`mermaid-lint.yml`	pull_request, path-filtered to `**.md` excluding `REVIEWS.md`; workflow_dispatch	neither (advisory)
`dev-proxy-wheel.yml`	push to develop, path-filtered to the proxy binary's inputs; workflow_dispatch (dry-run by default)	neither
`codeql-analysis.yml`	push, pull_request, weekly	neither
`prune-reviews.yml`	push to develop	neither
`pin-indirect-dependencies.yml`	daily, and on PRs touching the pinning script	neither
`pypi-storage-check.yml`	weekly (Monday 06:00 UTC), workflow_dispatch	neither

rust.yml is advisory rather than gating. Rust breakage still blocks merges, because the proxy wheel is built by the direct-qemu and sf-e2e lanes in the smoke tier; what never runs against the merged tree is clippy and cargo test, which is an accepted gap.

The direct-qemu lane also runs nightly, because the merge queue does not re-run it against the merged tree.

mermaid-lint.yml renders every tracked markdown file that contains a mermaid fence and fails on any diagram that does not parse. It exists because mermaid fails at render time rather than at commit time: a syntax error commits cleanly, passes every other linter here, and then shows an error box on GitHub and nothing at all on the mkdocs sites. It lints the whole corpus rather than the changed files, because a mermaid version bump can break a diagram no diff touches. The script and the workflow are byte-identical copies of templates/mermaid-lint/ in shakenfist/development, pinned mermaid-cli tag included; sync from there rather than editing either in place, so drift shows up as a diff. Rendering runs through puppeteer and needs a browser, hence a container, hence the only job here on debian-13-docker — a label that must also appear in .github/actionlint.yaml or actionlint fails on the workflow.

It is advisory, and must not be made a required status check. It is path-filtered to markdown, and a path-filtered workflow that the ruleset requires never reports on a pull request that touches no markdown, blocking that pull request forever — the same trap the check_paths filter jobs exist to avoid, and the one tools/check-required-checks.sh catches after the fact. Adding merge_group: to its triggers is no better: paths is not supported on that event, so every merge would spin a virtual machine to re-lint diagrams the pull request already linted. If the lane ever needs to gate, the way to do it is to give a job the gate already covers a docker-capable runner and add tools/mermaid-lint.sh there as a step.

demo-compose.yml and mermaid-lint.yml are the two lanes that need a container runtime, and they get one in different ways. demo-compose.yml runs on debian-13, whose image does not have one, so it installs Docker Engine itself via tools/demo/install-docker.sh — from Docker's own apt repository, which is what Docker's own install instructions assume and what the lane has always used. Trixie would now serve too, packaging Engine 26.1.5 and Compose 2.26.1 as a docker compose plugin where bookworm packaged neither, so the choice is convention rather than necessity. The script is idempotent, so adding docker to the runner image — or installing the distribution's packages into it — would make it a no-op rather than a conflict. It also configures the daemon and builds for the runner's proxy, which neither inherits on its own.

The demo lane installs the newest dev kerbside-proxy wheel published from develop rather than building the proxy, except when that wheel speaks a different gRPC contract from the checkout, as it does for any change to kerbside/rpc/kerbside.proto until the change merges and a new dev wheel is published. tools/demo/pair-proxy.sh compares the published binary's --contract-hash with the checkout's proto and, when they differ, builds the proxy from the checkout and passes it to the image build as KERBSIDE_PROXY.

Two behaviours only matter when driving CI by hand: on a workflow_dispatch run of functional-tests.yml an unselected target skips cleanly via a job-level if: (it does not report red), and instance readiness in the shakenfist/actions provisioning playbook gates on cloud-init completion, not just an open SSH port.

Gate jobs and required checks¶

Branch protection cannot require "whatever ran"; it requires named checks. Since a job that does not run reports nothing, and a required check that never reports blocks every merge forever, each tier ends in a small aggregate gate job whose only work is to assert that everything it depends on succeeded or skipped:

Check	Asserts
`Can see status`	nothing; it always succeeds, proving the workflow was evaluated at all
`Can enqueue`	the smoke-tier jobs in `functional-tests.yml`, including the credential scan, on non-merge_group events
`Can enqueue: direct-qemu`	the direct-qemu lane
`Can enqueue: sf-e2e`	the Shaken Fist end-to-end lane
`Can merge`	`sanity_checks`, the docs checks and the credential scan, on merge_group events only

Those five names are the entire required-check list on the develop ruleset. Because a skipped required check satisfies the rule, one list serves both refs: on a pull request Can merge skips, and in the merge queue the three Can enqueue checks skip.

The two smoke workflows cancel a whole run when a newer push supersedes it, and their gates run under !cancelled() rather than always(), so a superseded run's gate reports cancelled instead of red. A gate skipped that way reports cancelled, not skipped, so it still blocks a head whose run someone cancelled by hand.

The binding between a required check and the job that satisfies it is the job's display name, matched as a string. Renaming a gate job without updating the ruleset blocks every merge in the repository. sanity_checks runs tools/check-required-checks.sh to catch that as a red smoke check rather than as an outage: it asserts every required context in the exported ruleset (.github/exported-config/ruleset-*.json, archived daily by export-repo-config.yml) still matches a job name in .github/workflows/.

Documentation checks¶

docs_checks runs tools/check-backend-tls-claims.py, which fails when a sentence in docs/use-cases/ or the Use Cases table in docs/index.md claims the Kerbside-to-hypervisor leg is TLS'd or certificate-pinned without naming the condition that makes it so. Five consecutive phases of PLAN-use-case-docs.md overstated that leg and three were caught by review rather than by writing, so the check exists to trip on the sixth.

It is its own job, and that is the point. sanity_checks is gated on check_paths, whose filter excludes docs/**, so a documentation-only pull request skips it entirely -- which is every pull request this check exists to police. It lived there for one review round before that was noticed. docs_checks is ungated for the same class of reason credential_scan is.

It is a tripwire and not a proof, and its limit is worth stating concretely rather than as a caveat. It works on blocks -- a bullet, a paragraph or a table row -- and any conditional word anywhere in the block satisfies it. So it catches a bald claim in a block of its own, which is what the first of those five phases wrote, and it does not catch a claim mixed into a block that names some other condition. Measured against the real case: the version of docs/use-cases/shakenfist.md that said Kerbside "pins the certificate subject the node publishes" -- false when the node publishes none -- passes this check, because the same bullet correctly conditions the TLS escalation on NEED_SECURED.

That makes the conditional vocabulary load-bearing, because a word in it that does not actually name a condition exempts every block it appears in. Review found configured doing exactly that, on pages that say "configured cloud" and "configured source" constantly, and it was dropped along with should and none. Anything added back needs a case in kerbside/tests/unit/test_check_backend_tls_claims.py that fails without it. That file is the reason a regex edit here cannot quietly neuter the check: it asserts the known-bad sentences still fail and the current tree still passes. tools/mutate-backend-tls-claims.py is what proves that file -- it breaks the guard once per rule and asserts the suite notices each one, so a vocabulary change is only covered once a mutation for it fails without the test. Four of its mutations exist because a run reported them uncaught.

It is a manual proof and no CI lane runs it, which matters because the point of the section above is that a claim of enforcement should name where. Run it by hand whenever you change the guard's vocabulary or its matching, and add a mutation for the rule you changed. What CI enforces is the unit test; the mutation tool is how you find out whether that test would have failed.

A green run means nobody wrote a careless sentence, not that the prose is right. The conditions themselves are in rust/kerbside-proxy/src/backend.rs: TLS happens when the hypervisor rejects plaintext and a secure port is configured, the subject is pinned when the source supplied one, and a private CA has to be supplied or the handshake fails against the public web trust store.

Merge queue concurrency¶

Every job in functional-tests.yml carries a concurrency group so a superseded run is cancelled rather than left to finish. Keying that group needs care, because github.ref means something different in the merge queue: it is the per-attempt queue branch, gh-readonly-queue/develop/pr-NNN-SHA, and GitHub mints a fresh SHA each time it rebuilds the group — which it does on every push to develop. Keying on it puts each rebuild in a group of its own, so nothing ever matches and nothing is ever cancelled. The superseded runs keep building whole clouds against sfcbr, starving the one group that can still merge.

So on merge_group the group is keyed on the base branch instead, and on every other event on github.ref as usual. The two cloud lanes, which no longer run in the queue, key on the event as well as the ref, so a dispatch on develop -- the pre-tag check -- and the nightly never cancel one another; a nightly cancelled that way would file no tracking issue. That is only safe because the develop ruleset sets max_entries_to_build: 1: the queue builds one entry at a time, so any other in-flight merge_group run is by definition superseded and GitHub has already abandoned its queue branch. Raising max_entries_to_build above 1 would make this wrong — speculative groups for different entries would then cancel each other — so that setting and this concurrency key have to move together.

Only a run's first attempt takes the base-branch key. A re-run keeps its original merge_group event, so with that key a "Re-run jobs" on an old queue run (see Releasing for when a release needs one) would cancel the live queue entry or be cancelled by it. A re-run therefore keeps its own queue branch's group.

The sibling smoke workflows (direct-qemu-functional.yml, sf-e2e-functional.yml) still key on github.ref alone. They trigger on merge_group but skip their heavy jobs there, so a piled-up run costs seconds and no cloud capacity.

Fixing it here was not enough on its own, because sfcbr is shared. A merge group in this repository has failed to place its oVirt instance while two superseded shakenfist/shakenfist merge groups — the same defect, in a neighbouring repository — held the capacity. The pattern is now a fleet consistency audit, merge-group-cancellation, which checks every repository whose workflows can run on merge_group, and every reusable workflow that inherits such an event.

For the design rationale see plans/PLAN-two-tier-ci.md.

The credential scan¶

The credential_scan job runs gitleaks over every commit reachable from HEAD — on a pull request that is the branch under test plus all of develop — and fails the build on any finding. It is the only job in either tier which is not gated on check_paths: every other lane can skip a documentation-only change, but a credential pasted into a code sample is a credential, and the review notes under .vscode/ are prose the rewriting pre-commit hooks deliberately leave alone so that a content scanner can read them.

Run it the same way CI does:

tools/gitleaks-scan.sh                     # gitleaks on $PATH
tools/gitleaks-scan.sh --gitleaks /tmp/gitleaks

The script does two things, and the second matters more. It scans, and before scanning it plants an SSH private key and a JWT in a scratch directory and fails unless gitleaks reports both. A scan that finds nothing is otherwise indistinguishable from a scan that cannot find anything — a shallow clone, a broken rule, an allowlist that has grown until it forgives everything. Those two shapes are the ones Kerbside actually handles: private keys move through the CI lanes and the deployment tooling, and the API authenticates callers with JWTs.

Two flags in the job are load bearing. fetch-depth: 0, because a secret committed and then reverted is still in the history and still needs rotating; the script refuses to run against a shallow clone rather than report a clean history it never looked at. And --log-opts="HEAD", because the gitleaks default is to scan every ref, which on a repository that publishes a site from a branch turns seconds into minutes of duplicate findings misattributed to unrelated merge commits.

gitleaks is downloaded with a pinned version and sha256 rather than installed from apt, because the version has to be ours to choose: .gitleaks.toml is written against 8.16's schema, in which per-rule allowlists are a single [rules.allowlist] table rather than the repeatable array later releases and the current upstream documentation describe, so an archive that moves under us breaks the configuration. The shared static pool granting no passwordless sudo is a second reason. To move the pin, run tools/gitleaks-scan.sh against the new version locally first and check that the positive control still passes.

Accepting a finding¶

History cannot be rewritten to unpublish anything from a public repository — the objects survive in every fork — so accepting a finding is a claim that the credential has been revoked where it was trusted, not that it has been tidied out of sight. Never suppress a finding for a credential that still authorises something.

There are two mechanisms and they are not interchangeable:

Content that recurs — a documentation placeholder, a test fixture, an upstream default — goes in an [allowlist] regexes entry in .gitleaks.toml, keyed on the text. Editing the paragraph around a placeholder produces a new finding in a new commit, so anything keyed on a commit would need replacing every time. Prefer a regex to a path: blinding a whole file also blinds a real credential added to it later.
A specific historical event goes in .gitleaksignore as a commit:path:rule-id:line fingerprint, which forgives that one occurrence and nothing else — the same secret in a new commit fails the scan again. Comment each entry with what the credential was and what was done about it; an undocumented entry is indistinguishable from a mistake.

The history is clean as scanned, so neither file carries an entry today and .gitleaksignore does not exist yet.

One known gap: the SPICE console token minted by kerbside/consoletoken.py is 48 characters of unadorned base62, which no regex can tell apart from any other identifier of that length, so a leaked one would not be caught. Giving it an identifying prefix — the way GitHub uses ghp_ and Shaken Fist uses sfk_ — would make it scannable, but it is a wire-format change to a value SPICE carries as a password. Issue #357 tracks it.

This is distinct from GitHub's own secret scanning, which detects known third-party credential formats and needs GitHub Advanced Security for custom patterns. The fleet audit behind this lane is secret-handling.

End-to-end CI coverage¶

The proxy is exercised end to end in CI: the direct-qemu functional lane boots a real qemu/SPICE guest, drives it with the ryll headless client through the proxy, and asserts the full Sextant scenario, plus API-driven in-flight session termination and a non-gating relay-latency loadtest. See proxy-architecture.md and ARCHITECTURE.md. The same proxy path can be exercised locally without MariaDB or the daemon via the standalone mock harness — see direct-qemu-harness.md.

Ryll¶

Ryll is the upstream Rust SPICE client at shakenfist/ryll. The latency loadtest image builds the Ryll binary from source (stage 1 of loadtests/latency/Dockerfile) and ships it in the runtime stage. A Python orchestrator at loadtests/latency/orchestrator.py drives Ryll's control socket and writes a CSV of latency samples.

The metric measured is keypress-to-screen latency: the time between the orchestrator sending a send_key down and the first surface_drawn event that follows it, paired FIFO. The orchestrator hard-fails if the client does not advertise and accept a surface_drawn subscription (control socket v1.1 or newer), rather than silently falling back to a different metric under the same column name. The CSV is one float per line, seconds, no header.

Shaped-link latency rig¶

tools/shaped-link/ measures the same keypress-to-draw latency on a deliberately slow link, for performance work rather than CI. It runs qemu, the proxy (started with tools/direct-qemu/'s scripts) and Ryll in unprivileged network namespaces with tc netem shaping between client and proxy, across a matrix of link profiles, guest activity levels and proxy settings, and summarise.py renders the results as a table.

Its metric differs from the loadtest's. With screen activity running, the first draw after a key press is almost always an activity frame, so keydraw-latency.py pairs each press with the first draw that touches the guest's key box instead. That needs a rect field on surface_drawn which the control socket protocol does not have yet; build-ryll.sh patches it in, and builds ryll at a fixed commit (RYLL_REF, default 0d140b3) because the patch no longer applies to ryll develop. What the rig has measured, and how to rerun it, is in performance/proxy-backpressure.md, performance/streaming-rebaseline.md and performance/ack-window.md. The header of run-matrix.sh documents every variable.

Beyond latency, the rig can measure the transcoding display channel against the relay:

Proxy flags per config. A CONFIGS entry takes two optional fields after its socket settings, name:lowat:rcvbuf:flags:displays. flags are extra proxy flags separated by commas, each --flag or --flag=value; displays limits the config to the named DISPLAYS entries. For example, with DISPLAYS naming off and all displays, tx:0:0:--transcode-display:off runs the transcoder only with streaming off, since it never streams whatever qemu allows. The proxy must be built with make -C rust/kerbside-proxy release FEATURES=transcode.
CPU and memory. Every case samples the proxy's and qemu's CPU time and RSS from /proc into proc.txt, every 0.5 s. summarise.py reports cores used over the key presses, mean and peak RSS, and a transcoded channel's pixmap cache peak from the proxy's log.
Image quality. QUALITY=1 adds screenshots, taken through Ryll's control socket screenshot verb, which returns a lossless PNG of what the client holds. Once every latency sample is taken, capture-quality.py saves mid.png while the guest's activity still runs, then presses F12. The keydraw guest (kd.frames, rebuilt with build-guest.sh) then draws its final activity frame and stops. Frame n is a function of n alone, so that picture is the same in every run. After QUALITY_SETTLE_MIN seconds (default 2, longer than the transcoder's 500 ms lossless delay) it waits for two identical screenshots a second apart, with nothing unacknowledged on the proxy's client sockets, and saves the last as frozen.png. An activity that froze on its own before the last key press (QUALITY_FREEZE_FRAME, default 36000 frames) marks the case froze-early, and its latency samples are dropped.
Scoring. quality.sh cases --reference REF.png WORKDIR scores every case's screenshots with quality.py and writes quality.json beside them; summarise.py then adds a quality table. The reference is the frozen.png of an unshaped (PROFILES=0:0), relayed case with streaming off, where spice-server sends only lossless draws. Take it twice and check with cmp that the two are byte-identical before believing any comparison. mid.png is compared over the static background only, which should match exactly; frozen.png over the whole frame, by SSIM and PSNR. The guest's key box is left out, because its colour depends on how many presses arrived.
The venv. quality.py needs numpy and Pillow, which nothing else in Kerbside does, so quality.sh keeps them in a venv of their own at QUALITY_VENV (default ${XDG_CACHE_HOME:-$HOME/.cache}/kerbside/shaped-link-venv), made on first use and safe to delete. Because the script executes that venv's python, it refuses one whose directory or interpreter this user does not own, and creates a new one mode 700 and only under a parent directory no other local user can write -- so a QUALITY_VENV set by hand needs to be a path you own. quality.sh test runs test_quality.py.

The rig can also put a SPICE server other than qemu's own in front of the guest, and change the guest's display device:

A SPICE helper on qemu's D-Bus display. A DISPLAYS entry whose streaming field is helper runs qemu with -display dbus,p2p=on and no -spice, and puts andris in front of it with its qemu platform (--platform qemu --qmp <socket>). andris listens on the port and ticket spice-server would have used, so the mock control plane and the proxy see an ordinary SPICE server. With RYLL_BIN and the rest set as for any run:

tools/shaped-link/build-andris.sh /tmp/sl   # andris at ANDRIS_REF, in Docker
Q=qemu-system-x86_64 G=/tmp/sl/guest
ANDRIS_BIN=/tmp/sl/andris/target/release/andris \
    DISPLAYS="off:$Q:off:$G helper:$Q:helper:$G" CONFIGS=relay:0:0 \
    tools/shaped-link/run-matrix.sh

build-andris.sh builds from a sibling ../andris checkout when there is one (ANDRIS_SRC overrides it), at a pinned commit (ANDRIS_REF), with make release FEATURES=qemu in andris's own Docker image. ANDRIS_ARGS passes extra andris flags. A helper case adds andris's CPU time and RSS to proc.txt as a line of its own, so summarise.py reports qemu and andris separately. It also writes andris.log, and an andris-died marker if andris exited mid-case, which counts like qemu-died. On the backend leg, the ss columns that name qemu then describe andris's socket. - std-vga. GUEST_VGA=std gives every display in the run qemu's std-vga in place of virtio-vga. The guest drives it with bochs-drm, which build-guest.sh includes. std-vga has no DIRTYFB, so qemu finds what changed by polling its dirty bitmap every 30 ms.

Testing the SPICE console of an oVirt VM¶

tools/test-ovirt-console.py is Kerbside's oVirt SPICE console probe. It connects to the oVirt engine API, finds the booted test VM (by default any VM named smoke-test-*), checks that SPICE display is configured, and performs a SPICE protocol handshake against the console port. This is the Kerbside-specific check and lives here because we iterate on it alongside the proxy.

python tools/test-ovirt-console.py \
    --url https://ovirt-engine.example/ovirt-engine/api \
    --password secret \
    --ca-file /path/to/ca.pem

The generic plumbing it builds on lives in the shakenfist/actions repo, which CI checks out alongside this one:

tools/start-test-target.py — generic oVirt smoke test: sets up a datacenter, cluster, hypervisor host, and local storage domain, uploads a disk image, and boots a VM (smoke-test-*, SPICE display by default) to prove the deployment works. test-ovirt-console.py then probes the VM it creates.
tools/ovirt-install-base.sh — base package installation (EPEL, utilities)
tools/ovirt-patch-ovn.sh — patches oVirt 4.5 OVN Ansible role bug (#949)
tools/ovirt-prepare-host.sh — engine health check, SSH setup, KVM verification
tools/ovirt-gather-artifacts.sh — collects RPM lists and logs for CI artifacts

The lane's own tools/ovirt-e2e/gather-diagnostics.sh runs on the runner and drives that last script. It exists because the script is delivered to the guest several steps into the job, so a run which failed during provisioning had nothing to run, exited 127, and — under bash -e — uploaded no artifacts at all. When the guest-side bundle is unavailable the wrapper collects a fallback set over ssh instead: cloud-init state and logs, the sshd journal, the interpreter and OS versions, disk and memory. It always leaves a bundle behind, with a gather.log recording every command and its exit code, and it exits zero by design — the runs where it matters are the ones that have already failed.

The direct-qemu lane¶

The direct-qemu lane publishes a Can enqueue: direct-qemu gate job, which is a required status check; see Gate jobs and required checks.

How the lane is put together¶

The lane runs qemu, Kerbside and ryll as plain processes on the runner rather than in containers: the self-hosted runner is itself a throwaway VM, which is already the isolation boundary, and passing /dev/kvm into a container would add privileged-container setup without adding isolation. The Uncalibrated Sextant guest is committed as tests/fixtures/uncalibrated-sextant.qcow2 (it is well under a megabyte, so plain git rather than LFS), which keeps the lane hermetic and makes a change of guest visible in a pull request's diff; tools/direct-qemu/rebuild-sextant-qcow2.sh refreshes it. TLS material is minted fresh by generate-tls.sh on every run, so no key is committed and no certificate can expire under the lane.

Sextant is a bare UEFI application, which constrains the qemu command line in start-qemu.sh. It needs -vga qxl, without which spice-server offers no display channel; its disk must be on the q35 machine's default AHCI controller, since a bare UEFI application has no virtio block driver, which in turn rules out -nodefaults; and the lane prefers the 4M OVMF build, the one Sextant's own release check boots.

Ryll is built from its default branch on every run, so the lane's clients check only the control socket's major protocol version: a check on the exact minor version failed every run the day ryll's protocol moved from 1.0 to 1.1. The Sextant scenario test provisions nothing itself: the lane boots the guest and starts Kerbside and ryll, and the test only reads the control socket and serial log paths it is given, so a lane on another platform can run the same test by providing the same two things.

The proxy wheel is installed the way a deployment installs it¶

The direct-qemu Rust leg builds and installs the kerbside-proxy wheel into the kerbside venv (install-proxy-wheel.sh → build-proxy-wheel.sh --native), so find_proxy_bin() resolves it via shutil.which on PATH — the real install path, which is what gives this lane its coverage value. start-kerbside.sh pre-checks the proxy binary through find_proxy_bin() for the same reason.

One ticket mint per session¶

db.session_ticket() holds SELECT ... FOR UPDATE on the session's token row while it mints, so that the channels of one session mint a backend ticket once between them. SQLite ignores FOR UPDATE, so the unit tests cannot show it. Straight after the lane's MariaDB is set up, run-ticket-lock-test.sh runs kerbside/tests/functional/test_session_ticket_lock.py against it: eight threads, released together by a barrier, call session_ticket() for one session with a mint that takes a second, and exactly one mint must serve all eight, both for an empty row and for a stored ticket inside the reuse margin. The test drops and recreates its own database, kerbside_locktest, and builds the schema with the migrations, so the daemon's kerbside database is untouched. It is skipped unless KERBSIDE_TEST_SQL_URL is set, and runs locally against any MariaDB the URL's user may create that database on:

KERBSIDE_TEST_SQL_URL=mysql://kerbside:kerbside@127.0.0.1/kerbside_locktest \
    python -m unittest -v kerbside.tests.functional.test_session_ticket_lock \
    kerbside.tests.functional.test_column_widths

Values fit their columns¶

SQLite ignores a VARCHAR's length, so a value too long for its column passes every unit test and then fails on MariaDB with "Data too long", or is silently cut short without strict mode. The same script runs kerbside/tests/functional/test_column_widths.py, in the same database and the same way, which stores a console under the longest uuid the Proxmox VE driver produces (a nine digit vmid and an SMBIOS UUID, 46 characters), with a console token and an audit event for it, and a source with a 100 character username, and reads each back unchanged. kerbside/tests/functional/sql_database.py holds the setup both modules share.

Live termination¶

verify-terminate-live.sh (Rust leg only) runs on an isolated lane: it POSTs the REST terminate endpoint and asserts the in-flight connection drops, via the proxy log line session terminated by control plane. This exercises the DB→ProxyControl bridge end to end rather than the mock.

SPICE password rotation¶

verify-ticket-rotation.sh runs on an isolated lane too. A source that mints a SPICE password per session replaces the password qemu holds each time it mints, and lets it expire, so the lane checks what that does to a client already connected through the proxy. Over the QMP socket lane-up.sh gives its qemu (start-qemu.sh --qmp-socket, mode 0600, driven by qmp-client.py) it calls set_password with a new password and no connected argument, as oVirt and Proxmox do, and then expire_password with now. After each, and a ten-second settle, the first session must still be up: its ryll still running, its control socket still reporting a surface and a screenshot, and qemu's query-spice listing exactly the channels it listed before, down to each connection's client port. A fresh session through the proxy, which the lane's static source still gives the original ticket, must be refused: the proxy logs hypervisor connection failed with PermissionDenied for its backend leg. Last, as a positive control, a ryll connected straight to qemu with the new password must get a display once the expiry is lifted. That runs last because spice-server drops its current client when a new one links. The new password lives only in an owner-only file in the lane's working directory; it is never on a command line or in a log.

Backend host subject checks¶

run-host-subject-checks.sh runs on the Rust leg after the main lane is torn down, since it reuses the same ports. It drives the standalone harness verify-rust-proxy.sh with BACKEND_TLS=1: qemu gets a SPICE TLS listener with a server certificate minted by generate-tls.sh (subject C=US,O=Kerbside CI,CN=qemu-hv, with no subjectAltName, as real SPICE certificates usually have none) and tls-channel=default, which is QEMU's spelling for TLS required on every channel. The plaintext port therefore answers need_secured, and the proxy reaches the TLS port through its real retry path. Two modes each run in their own working directory, with ryll headless as the client:

match: the mock control plane pins the correct host_subject. The session must be authorized and relay bytes in both directions, and the mock's audit log must hold no Hypervisor connection failed event. This mode is also the gate on the firewall allowlist (assert-firewall; see the direct-qemu harness).
mismatch: the mock pins CN=not-the-hypervisor. The session must be authorized, since authorization comes before the backend connect, but relay nothing; the mock's audit log must hold a Hypervisor connection failed event naming NotValidForName, and the proxy's log its pinned host_subject refusal line.

The rules being tested are described in Backend host subject pinning.

Tunnelled backend connections¶

run-tunnel-checks.sh runs on the Rust leg after the host-subject checks. A Target with a tunnel makes the proxy reach qemu's TLS port only through an HTTP CONNECT proxy, so the lane runs connect-proxy.py, which stands in for Proxmox's spiceproxy: it accepts a CONNECT only when the request line and the Host: header both name the expected connect host and port, relays it to qemu's TLS listener, answers anything else 401, and writes counts of tunnels opened and requests refused to a status file. The mock hands out a hypervisor_ip of 192.0.2.1 (TEST-NET-1, which nothing answers), so a session that comes up can only have come through the tunnel. Three modes each run in their own working directory:

match: the correct host_subject. The client gets a session, and the CONNECT proxy opened at least one tunnel per relayed channel and refused none.
no-subject: no host_subject. The session fails, the mock's audit log holds a Hypervisor connection failed event naming host_subject, and the CONNECT proxy was never asked for anything. This proves a tunnel without a pin is refused and that the refusal is audited.
mismatch: a wrong host_subject. The session fails at the TLS handshake through the tunnel.

The connect host is a random sentinel per mode, kept in an owner-only file so it is never on a command line. Afterwards every collected artifact is searched for each mode's sentinel, along with a copy of the script's own output, which is what the public step log shows. A file holding one is removed and the run fails, because the connect host is a credential that must not reach a log.

The latency loadtest¶

Both legs run run-loadtest.sh (non-gating, continue-on-error). It drives loadtests/latency/orchestrator.py to sample keypress-to-screen latency — real send_key events timed against the surface_drawn they produce — through the leg's proxy, and records p50/p95 as an artifact; the Python-versus-Rust comparison is read off the two legs.

It boots the purpose-built tests/fixtures/uefi-latency-guest.qcow2, which repaints on every keypress, rather than the Sextant scenario fixture, which leaves its Awaiting screen on the first key and freezes at the bootloader prompt. Like verify-terminate-live.sh it brings up its own isolated lane (separate WORKDIR, QCOW2 overridden) and tears it down before the shared scenario lane starts.

This is distinct from the local mock harness (direct-qemu-harness.md), which needs no daemon and no database.

The Shaken Fist end-to-end lane (`sf-e2e`)¶

.github/workflows/sf-e2e-functional.yml is the only lane that exercises the type: shakenfist console source against a real cluster. It stands up a single-node Shaken Fist with shakenfist/actions/build-smoke-cluster, whose deploy also deploys a co-located Kerbside with a type: shakenfist source: the deploy profile tools/sf-e2e/deploy-profile.yml.j2 installs this pull request's ansible/ collection and hands its kerbside role this pull request's Kerbside and proxy wheels. The cluster is deployed twice, and the second deploy must restart no sf-* or kerbside-* unit. The lane then drives an SF-minted token through offline verification, exchange, and a proxied SPICE session against the Sextant guest booted inside the SF instance — followed by an adversarial matrix covering replay, expiry, wrong audience, unknown kid, and cross-namespace mint.

Driver scripts live in tools/sf-e2e/ (see tools/sf-e2e/README.md). It is a smoke-tier PR gate, and also runs nightly and on dispatch. Two things run it against develop: the 02:30 cron, and the Shaken Fist conductor's periodic tier, which dispatches it on develop -- sf-e2e is the only lane here the conductor dispatches, which is why the other lanes' failure-issue jobs are cron-only. A failing run against develop files or updates the "Nightly sf-e2e lane failing" tracking issue, because nothing else runs the lane against develop itself and a red entry in the Actions tab alerts nobody. A dispatch on any other ref is the interactive debugging case: it says nothing about develop's coverage and files nothing, and nor does a pull request, whose red is already visible there.

Every run uploads an artifact bundle with the Kerbside unit journals and the ryll, SF daemon and libvirt logs, redacted before upload; the README's Security section lists what is collected.

The exchange needs a real cluster because it cannot be faked in the cloud-free direct-qemu lane. Only a type: shakenfist source's keys can verify a token, and the exchange then requires the console to belong to that source; a console seeded into the database by hand is removed by the daemon's next scrape, which reaps every console the source did not report; and Shaken Fist reads KERBSIDE_URL when its API starts, so the setting has to be in place at deploy time. Kerbside runs on the cluster's own node and reaches the API on localhost, so the lane need not expose the cluster's API beyond the node. The offline verification itself is covered case by case by the unit tests in kerbside/tests/unit/test_sf_token.py and test_api.py.

The oVirt end-to-end kerbside lane¶

The ovirt_matrix job in .github/workflows/functional-tests.yml does more than build and probe the oVirt environment: it also deploys the PR's own kerbside (package plus the manylinux Rust proxy wheel) on the CI runner, registers a live type: ovirt source against the engine it just built, and relays a real SPICE session from the oVirt hypervisor through the Rust proxy — asserting from the proxy log that the backend leg escalated to TLS with a non-empty certificate-subject pin on every escalation, then terminating the in-flight session via the REST API and asserting the proxy dropped it.

The engine also holds a second VM, no-spice-test: diskless, network-boot, with a VNC display and therefore no SPICE console (tools/create-ovirt-vnc-vm.py). It exists to be ignored. Discovery has to skip a VM it cannot broker and carry on scraping, and every other VM in every lane has a SPICE display, so that branch had never run in CI — which is how a missing continue in kerbside/sources/ovirt.py survived: it errored the whole source, dropped every VM discovered after the offending one, and reaped their consoles as no longer available, once a minute. drive-console.py now asserts that VM is absent from the console list while the SPICE one is present.

Attaching that VM's NIC has one trap worth knowing about. The lane runs two datacenters — Default, from engine-setup, and test, from start-test-target.py — and each gets its own network named ovirtmgmt, with its own id and its own vNIC profile of the same name. Selecting a profile by name alone picks whichever the engine lists first, and attaching the wrong datacenter's profile fails with HTTP 409 The specified Logical Network doesn't exist in the current Cluster. create-ovirt-vnc-vm.py therefore resolves the network through the cluster that will host the VM, which is the constraint the engine actually enforces; _resolve_vnic_profile is covered by kerbside/tests/unit/test_create_ovirt_vnc_vm.py.

That generalises: anything that looks up an oVirt object by name must scope the lookup to the test cluster or its datacenter (issue #283). A bare name match fails only when the engine happens to list the wrong one first, and this code runs in the nightly tier only — so a smoke-green PR proves nothing about it. Dispatch the oVirt lane on a pull request that touches it.

Before it builds the environment, the lane asks the under-cloud to cache the guest's rocky:8 base image (tools/ovirt-e2e/precache-image.py). Each run's throwaway namespace fetches the 1.9 GB image afresh, and when that fetch started only with the instance it raced the six minutes Shaken Fist gives an instance start to wait for its dependencies: the instance never left initial, which read as an under-cloud capacity problem (#284) but was a slow download.

The runner-side scripts live in tools/ovirt-e2e/ and are documented in tools/ovirt-e2e/README.md. The lane is a worked example of the deployment described in use-cases/ovirt.md, which is the operator-facing version of what it proves.

Log-derived oracles¶

drive-console.py asserts against the proxy's log text. It strips ANSI before matching, and keeps "the field would not parse" separate from "the field was empty" — the first is a harness fault, the second is a real unpinned TLS leg. Conflating them (issue #272) reported a broken parser as a security failure for two days. Any new log-derived oracle should draw the same distinction. Proxy log colouring is described in proxy-architecture.md.

The Proxmox end-to-end lane¶

proxmox-functional.yml proves the type: proxmox source (kerbside/sources/proxmox.py) against a real Proxmox VE node. It deploys a single-node PVE 9 with shakenfist/actions' deploy-proxmox-on-shakenfist action, deploys the PR's own kerbside (package plus the manylinux Rust proxy wheel) on the CI runner, registers a type: proxmox source against the node, and relays real SPICE sessions through kerbside's proxy and PVE's spiceproxy. The proxy wheel and ryll are built before the node is booked, so a compile error never spends a node.

It runs nightly, on workflow_dispatch, and on pull requests that touch any of:

the Proxmox source: kerbside/sources/proxmox.py, base.py and minting.py, and the source-type registry in kerbside/sources/__init__.py, which maps type: proxmox to it;
kerbside/config.py, whose CONSOLE_TOKEN_DURATION sets how long the lane holds a session;
kerbside/rpc/**, which includes kerbside.proto;
the proxy's backend leg and relay, rust/kerbside-proxy/src/backend.rs and relay.rs, its Cargo.toml and Cargo.lock, which pin the ryll crate whose log lines the lane reads, and tools/build-proxy-wheel.sh, which builds the wheel the lane installs;
docs/console-sources.md, whose account block the lane runs;
the workflow itself and tools/proxmox-e2e/**;
the direct-qemu scripts the lane reuses: tools/direct-qemu/start-kerbside.sh, smoke-client.py, setup-mariadb.sh and generate-tls.sh.

kerbside/api.py and kerbside/db.py are exercised by the lane but left out of the filter: they change on most pull requests, and would book a PVE node for each. The nightly covers them.

It has no merge_group trigger and is not a required check: a path-filtered required check never reports on other pull requests, which would block them forever. So a pull request that breaks it can merge red, and the nightly Nightly proxmox lane failing issue is what catches that. Pull requests from forks skip it, because its runner holds the CI mesh's SSH key.

What it asserts, in short:

the account in docs/console-sources.md works: the lane runs that page's pveum block on the node (with the user renamed), and fails if the commands are wrong or contain anything but the four commands the account needs;
both of the account's privileges are needed, not only sufficient: tokens lacking VM.Audit and VM.Console are each refused with 403, and the documented token is refused status/stop;
discovery lists exactly the one SPICE guest, and skips guests with a std display, no SMBIOS UUID or a stopped state, and a template when PVE accepts a diskless one;
PVE's own access log shows the expected number of spiceproxy mints at each checkpoint: none after the scrape and the refused direct .vv, one once the first session's channels have connected and still one after it is held, two once a second session's have, and three after the lane's own mint and at the end. So a session's channels share one ticket, on real PVE;
the backend leg is tunnelled, TLS, and pinned to the node's subject, and a session survives past its 30 second ticket and CONSOLE_TOKEN_DURATION;
the second session displaces the first, because qemu's SPICE server serves one client at a time: the first session's ryll exits and the proxy logs each of its relays ending, while the second passes frames. Then the lane mints one ticket itself, as kerbside does, and the second session still passes frames, which shows on real PVE that a password change keeps a connected client. Terminating the second session over REST drops it;
no token secret, backend ticket, .vv password, API JWT, value from the lane's own mint or pvespiceproxy: reaches an uploaded artifact.

Like the oVirt lane, its oracles keep a harness fault apart from a defect (see "Log-derived oracles" above), and the mint count adds a third verdict, a timing fault, for a channel that connected too late to reuse its session's ticket. Reading each session's backend ticket can give a timing fault too, when the harness was too slow to read it before it expired.

The runner-side scripts, the topology, a diagram of the path proven, the mint-counting method, the security rules and what only the lane exercises are documented in tools/proxmox-e2e/README.md. Dispatch the lane on a pull request that changes the Proxmox driver in ways the path filter does not cover.

Tempest tests against a Kolla-Ansible deployment¶

The openstack_matrix job deploys an all-in-one OpenStack on a single Shaken Fist guest and runs Tempest against it. That guest is a debian:13 image, and the distro is load bearing rather than incidental: Kolla-Ansible master requires ansible-core>=2.20, every ansible-core 2.20 release requires Python 3.12 or newer, and kerbside-patches' tools/bootstrap-kolla-ansible installs Kolla-Ansible into a venv built on the guest's system Python. A guest below 3.12 fails during pip resolution, before anything is deployed. Debian 12 ships 3.11 and broke the lane this way (#426); Debian 13 ships 3.13. kerbside-patches tests every all-in-one master topology on debian:13, so this matches upstream rather than diverging from it. The containers Kolla builds are Debian trixie regardless of the guest, and are unaffected.

The tempest-plugin/ directory is a separate releasable that contributes Kerbside-specific Tempest tests; see tempest-plugin/README.md for what it covers.

tools/run-tempest-tests drives a curated subset of those tests against a running Kolla-Ansible deployment. It is invoked automatically by the openstack_matrix job in .github/workflows/functional-tests.yml after the test-console smoke check, so the GitHub Actions CI iterates on the plugin's tests every night (the cloud matrices run in the nightly tier, not per-PR) rather than relying on upstream Zuul as the first signal. The script:

Creates a Python venv at /srv/kerbside-tempest/venv.
Pip-installs tempest, python-tempestconf, and the local tempest-plugin/ checkout into it.
Runs tempest init plus discover-tempest-config against /etc/kolla/clouds.yaml's kolla-admin cloud with compute-feature-enabled.spice_console True.
Injects the [kerbside] group pointing at the Kolla CA bundle.
Runs tempest run against a regex that selects the kerbside plugin tests. The upstream tempest.api.compute.admin.test_spice (spice-direct) test deliberately bypasses Kerbside by connecting straight to the libvirt SPICE port, so it is not in the default regex — pass --regex to opt back in if you want it.

Run it manually on a deployed all-in-one node with sudo bash tools/run-tempest-tests; pass --help to see knobs (regex, workspace location, CA bundle path, etc.).

The job's log bundle comes from tools/gather-kolla-logs.sh, which runs kerbside-patches' tools/gather-logs on the guest and adds the Tempest logs. Like the oVirt lane's gather-diagnostics.sh, it always leaves a bundle and exits 0: when the under-cloud never created the guest it says so in gather.log and raises a warning, rather than adding two red steps to a failure that happened earlier.

Sextant scenario test (direct-qemu lane)¶

The plugin also contains an end-to-end scenario test at tempest-plugin/kerbside_tempest_plugin/tests/scenario/test_sextant_scenario.py. It drives an Uncalibrated Sextant UEFI guest through the full Awaiting → Booting → bootloader-ignore → paste → Parked → shutdown sequence over Ryll's control socket and asserts two independent oracles: the live digest_updated QR event stream (frame counters strictly increasing; per-beat record predicates) and the post-mortem serial drain (canonical ordered event subsequence, monotonic timestamps). The test requires ryll built with --features digest-decode (enabled automatically by the direct-qemu workflow).

Four [kerbside] tempest options support the scenario test: control_socket_path, serial_log_path, scenario_artifact_dir, and scenario_step_timeout (default 60 s). When control_socket_path is unset the test skips cleanly, so the plugin remains drop-in safe on the OpenStack lane. On the direct-qemu lane all four options are written by tools/direct-qemu/run-scenario.sh, which runs the test as the final (deliberately destructive) lane step — the final keypress causes Sextant to drain serial and ACPI-shutdown, terminating the guest and the ryll control socket. Screenshots are saved per beat into scenario_artifact_dir and uploaded as CI artifacts alongside tempest.log.

Build the load testing OCI container images¶

There are a series of OCI container images intended for load testing. These need to be built from the top level directory of the repository because of the way docker build likes to constrain what files you can copy into a container image.

Latency load test¶

This is the first load test that was implemented. It uses a UEFI binary as a test target and drives Ryll (the upstream Rust SPICE client) in headless mode against an OpenStack-provisioned instance. A Python orchestrator at loadtests/latency/orchestrator.py connects to Ryll via its control socket, sends spacebar keypresses every two seconds, and writes keypress-to-screen latency samples (each keypress timed against the first surface_drawn event after it) to a CSV, one float per line, in seconds. See the Ryll section above for the metric definition.

To build this OCI image, do this:

docker build . -f loadtests/latency/Dockerfile -t kerbside-latency:latest

For your convenience, there is also a version of this image at https://images.shakenfist.com/testimages/kerbside-latency.tar.gz

📝 Report an issue with this page