Proxmox source phase 3a: minting at connect time¶

This is the detailed plan for phase 3a of PLAN-proxmox-source.md. Read the master plan first; its What the measurements settled section, and within it What oVirt does, measured the same way and A second mint revokes the first, are assumed knowledge here.

Prompt¶

All work for this phase lands in shakenfist/kerbside. It changes no Rust, no protobuf and nothing in ryll: the Target message already carries a ticket, and this phase changes only where that value comes from. Consult ARCHITECTURE.md (particularly Session termination drops in-flight connections, which states the distributed deployment constraint this plan is built around), docs/proxy-architecture.md, docs/console-sources.md and docs/schema.md. Run tox -eflake8 and tox -epy3, and pre-commit run --all-files, before every commit.

Planning effort: high. The phase moves a credential's lifecycle, makes the authorize path call a remote API for the first time, adds a cross-node locking requirement, and touches a schema. Ticket and token lifecycle is named in PLAN-TEMPLATE.md as one of the invariants that pushes a step to high effort.

Scope¶

In:

A source-driver hook for minting a console ticket, and a registry mapping a source's type to its driver class, so the three places that dispatch on type today stop hand-rolling it.
Minting in AuthorizeConnection for sources that mint: one ticket per session, shared by every channel of that session on every proxy node, reused while valid and replaced when not.
Where that ticket lives between channels, and how it is removed.
ovirt.py requesting an explicit expiry for every ticket it mints, on both the proxied and the direct path.
ConsolesProxyVirtViewer no longer minting or storing an oVirt ticket, and a migration clearing the oVirt tickets already stored on Console rows.
Settling the supersession question the master plan left for this phase, by source reading and by a measurement in the direct-qemu lane.
The oVirt merge-tier lane proving a session outlives its ticket.
Documentation of all of the above, including the four places that describe oVirt's ticket lifetime incorrectly today.

Out:

Anything Proxmox-specific. No proxmox.py, no proto change, no tunnel. The hook is designed so that phase 4's driver fits it (see decision 3), but nothing here depends on phase 2 or on a Proxmox node.
The static source. Its ticket is operator configuration, not a minted credential, and stays on the Console row. Shaken Fist and OpenStack pass no ticket at all and are unaffected.
Two sessions against one VM. A second viewer's mint revokes the first viewer's password for new channels. That is how oVirt behaves today and it is recorded, not changed; see decision 8.
Encrypting Console.ticket at rest for static sources.
Issue #319 (the proxied .vv GET mints a token). This phase removes the oVirt ticket write from that GET, which shrinks the issue, but the token mint remains and the issue stays open.

What the survey found¶

Surveyed on 2026-09-28 against develop at 5475dab. The master plan's section for this phase was written on 2026-09-20, before phases 1b and 2 ran.

The master plan's storage decision does not survive the architecture. It holds the session's ticket "in the daemon's memory, keyed by session". Kerbside explicitly supports a load balancer spreading one session's channels across several proxy nodes (ARCHITECTURE.md:129-140, docs/proxy-architecture.md:469-477), and each node's daemon has its own memory. A channel landing on a second node would find no ticket, mint its own, and revoke the first node's: the per-channel race the master plan rejected, reintroduced across nodes. The shared database is the only bus every node can reach. Corrected at source in the master plan's What the measurements settled and its phase table, and in the index.md row; see decision 1.
A channel cannot arrive late. The master plan imagines "a usbredir channel on a device plug" arriving after the ticket ages out. AuthorizeConnection resolves the client's token with db.get_token_by_token(), which filters expires > now (db.py:498-507), and CONSOLE_TOKEN_DURATION defaults to one minute (config.py:234-236). So no channel of a session is ever authorized after its token expires, however long the session stays up. (spice-gtk opens its usbredir channels at session start and redirects devices over them later; it does not open a channel per plug.) The window in which a replacement mint could ever be needed is bounded by the token, which makes decision 2 possible. Corrected at source.
Supersession is settled by source, and it is benign. A new SPICE password is set by QMP set_password, whose connected argument decides what happens to connected clients and defaults to keep (qemu/qapi/ui.json:25-61). oVirt's engine builds the ticket request without an existingConnAction (ovirt-engine/.../SetVmTicketVDSCommand.java:34-47), so vdsm leaves the libvirt connected attribute unset (vdsm/lib/vdsm/virt/vm.py:5131-5132), libvirt passes no connected to qemu (libvirt/src/qemu/qemu_hotplug.c:4480-4485), and qemu keeps the clients. Proxmox calls set_password with no connected either (qemu-server/src/PVE/API2/Qemu.pm:3322). So a second mint revokes the first ticket for new channels only; established channels stay up. The same holds for expiry: ryll's Proxmox lane holds a session to mint+120 s against a password that qemu expires at +30 s, and passes. Step 3a.2 measures both on a real qemu anyway, because this is the property the design rests on.
oVirt's ticket lifetime is wrong in four places, in opposite directions. The master plan measured 7200 s, the engine default ovirt.py:148 takes by passing no expiry. But docs/use-cases/ovirt.md:90-94 calls oVirt tickets "short-lived", its limitations table (:288) says "roughly two minutes, set by oVirt", ARCHITECTURE.md:269 says the driver "acquires a short-lived graphics-console ticket", and tools/ovirt-e2e/drive-console.py:18-19 says they "expire in ~120s". After this phase kerbside sets the lifetime, so all four are rewritten in step 3a.8 rather than patched now.
Source dispatch is hand-rolled three times. main.py:169-180 builds a driver from source['type'], and api.py:420-430 and api.py:514-526 each construct oVirtSource directly. The authorize path would be a fourth. Decision 3 adds a registry instead.
The oVirt driver can hang. oVirtSource.__init__ fetches the engine CA with requests.get and no timeout (ovirt.py:46-48). That is tolerable in the scrape loop and in a .vv request. It is not tolerable in the authorize path, where a hang holds a gRPC worker (rpc/server.py:33, API_GRPC_WORKERS) and, under decision 1, a row lock that every other channel of the session is waiting on.
The tokens table is already per session. ConsoleToken (db.py:442-468) carries session_id, source, uuid and expires, one row per .vv issued. Every channel's authorize reads it. It is the natural home for a per-session credential, and its reaper (db.py:533-559, driven from main.py:283-287) already exists.
The proxied GET writes a credential. ConsolesProxyVirtViewer (api.py:482-534) mints an oVirt ticket and stores it on the shared Console row, where the next .vv for the same console overwrites it. Two concurrent sessions on one console therefore already share whichever ticket was minted last. That behaviour ends with this phase, because nothing writes the row for oVirt any more.
Nothing needs to change in Rust or the proto. servicer.py:129-141 fills Target.ticket from the console row. This phase fills it from the session ticket instead, for sources that mint. The proxy (rust/kerbside-proxy/src/session.rs:316-328) calls authorize once per channel with no deadline, and waits.

Decisions¶

The session's ticket lives on its token row, in the shared database. Two new nullable columns on consoletokens: backend_ticket (string) and backend_ticket_expires (integer, epoch seconds). The authorize path reads and writes them under SELECT ... FOR UPDATE on the token row, so exactly one channel of a session mints, on whichever node it lands on, while the others wait and then reuse what it wrote.

This is the decision a reviewer is most likely to dispute, because it puts a live console password in the database that the master plan said would stay out of it. The comparison that matters is not with memory, which does not work (survey finding 1), but with today: today an oVirt password sits on the Console row for two hours, shared between sessions, and survives the session. Under this decision it sits on one session's row for at most the token's lifetime plus a small slack (a minute and a half by default), and is cleared by the reaper once it is expired. The alternative of requiring sticky load balancing moves the constraint onto every operator and fails silently when they get it wrong.

Holding a row lock across a remote API call is deliberate. The call is bounded (decision 5), the lock is per session so nothing else contends for it, and the waiters want exactly the thing the holder is producing. An optimistic scheme (claim, mint, publish) would need its own timeout and recovery for a claimant that dies, which is the lock's job already.

The ticket is a secret field. ConsoleToken.export() leaves it out unless asked, mirroring CONSOLE_SECRET_FIELDS (db.py:246), and it is never logged.

An oVirt ticket's expiry covers the token, so oVirt never re-mints. When the first channel of a session mints, it asks for an expiry of the token's remaining lifetime plus TICKET_SLACK (30 s). The slack covers a channel authorized just before the token expires and then authenticating to qemu. After that no channel can be authorized (survey finding 2), so the ticket is never needed again. A deployment that lengthens CONSOLE_TOKEN_DURATION lengthens the ticket with it, which is what it asked for.
The hook is general, and phase 4 fits it without change. BaseSource gains a class attribute MINTS_TICKETS = False and a method mint_console_ticket(uuid, lifetime) that returns a MintedTicket(ticket, expires). A driver honours lifetime where its API lets it (oVirt) and reports the shorter expiry where it cannot (Proxmox's fixed ~30 s). The authorize path reuses a stored ticket while expires - now >= REUSE_MARGIN (10 s) and re-mints otherwise. For oVirt that branch never fires within a token's life. For Proxmox it fires for a channel arriving more than about 20 s after the first mint, which is safe by survey finding 3. A new kerbside/sources/__init__.py maps type to class (driver_class(source_type)) and replaces the three hand-rolled dispatches.
The direct .vv path keeps minting on request, with an explicit expiry of CONSOLE_TOKEN_DURATION. A direct .vv embeds the hypervisor ticket, so its lifetime has to cover the same user-paced gap a proxied .vv's token does. The operator has already said how long that gap may be. Reusing the setting rather than adding an OVIRT_DIRECT_TICKET_LIFETIME knob keeps one answer to one question. It replaces the 7200 s the path gets today by passing nothing.
A mint is bounded, and a failed mint denies the channel. The oVirt driver's CA fetch gets a timeout, and its SDK connection is built with one (MINT_TIMEOUT, 15 s for both). A mint that raises or times out returns Denied(reason='console ticket unavailable'), writes an audit event, and stores nothing, so the next channel tries again. The master plan already accepted that the source's API being down now breaks connection setup; this makes it break promptly and visibly.
Minting is audited, the ticket is not. A successful mint adds the audit event Minted backend console ticket with its lifetime in seconds; a failed one adds Backend console ticket mint failed with the exception's class name only. Reuse is not audited, since it happens once per channel and says nothing new.
Existing oVirt tickets are cleared by migration, and old daemons are not supported alongside new ones. The migration that adds decision 1's columns also sets consoles.ticket to '' for every console whose source has type = 'ovirt', removing up to two hours of live passwords at upgrade. A daemon from before this phase reads the (now empty) row ticket for oVirt and its channels fail authentication. Every node's daemon has to be upgraded together, which the shared-schema migration already implies; the release note says so.
Two sessions on one VM stay as they are. A second session's mint revokes the first session's ticket for new channels, and survey finding 3 shows its established channels stay up. The first session's remaining channels have all opened within a second or two of its main channel, long before a human opens a second viewer, so in practice nothing is lost. This is recorded in docs/console-sources.md rather than engineered around.

Step plan¶

One commit per step. Every commit passes tox -eflake8, tox -epy3 and pre-commit run --all-files on its own.

Step	Effort	Model	Isolation	Brief for sub-agent
3a.1	medium	sonnet	none	Registry. Create `kerbside/sources/__init__.py` with `driver_class(source_type)` returning `ShakenFistSource`, `oVirtSource` or `StaticSource` for `'shakenfist'`, `'ovirt'`, `'static'`, and raising `KeyError` for anything else (OpenStack has no driver; `main.py:175-178` skips it and must keep doing so). Import the driver modules lazily inside the function or at module level, matching how `main.py:20-22` imports them today, and check for an import cycle with `kerbside/sources/base.py`. Replace the dispatch at `main.py:169-180` with it, keeping its `openstack` skip and its unknown-type error. Do not touch `api.py` yet; 3a.4 and 3a.5 rewrite those call sites. Add `MINTS_TICKETS = False` and a `mint_console_ticket(self, uuid, lifetime)` that raises `NotImplementedError` to `BaseSource` (`kerbside/sources/base.py`), plus a `MintedTicket` named tuple `(ticket, expires)` there, with docstrings saying what the method promises: return a ticket valid until `expires` (epoch seconds), honouring `lifetime` where the source's API allows and reporting the shorter expiry where it does not. Unit tests in a new `kerbside/tests/unit/test_sources_registry.py`. Commit: "Look source drivers up by type."
3a.2	high	opus	none	Measure supersession on a real qemu. The direct-qemu lane's qemu has no monitor; `tools/direct-qemu/start-qemu.sh:121` sets SPICE up with `password-secret`. Add a QMP unix socket to that qemu (under the lane's existing work directory, mode 0600). Write `tools/direct-qemu/verify-ticket-rotation.sh`, modelled on `tools/direct-qemu/verify-terminate-live.sh`: bring up a ryll headless session through the proxy (as that script does), confirm its channels are up over ryll's control socket, then over QMP (a) `set_password` with protocol spice and a new password and no `connected` argument, and (b) `expire_password` with protocol spice and time `now`. After each, wait 10 s and assert over the control socket that the session is still connected and still has its surface. Then assert that a fresh connection presenting the old password is refused (the lane's mock or static source can be pointed at the old value; read how `verify-terminate-live.sh` and `mock-grpc-server.py:286` supply the ticket). Wire it into `.github/workflows/direct-qemu-functional.yml` as its own step beside `verify-terminate-live.sh` (`:185`). Keep the workflow step to one line calling the script. Never print a password: pass them through files or environment variables, not arguments that land in `ps` or the log. Record the result in this plan's Outcome. Commit: "Prove a new SPICE password keeps connected clients."
3a.3	high	opus	none	Schema. Use the `add-database-migration` skill in `.claude/skills/`. Add nullable `backend_ticket` (String) and `backend_ticket_expires` (Integer) to `ConsoleToken` (`db.py:442-468`) and its migration, with a working downgrade. In the same migration, clear `consoles.ticket` to `''` for every console whose `consoles.source` names a `sources` row with `type = 'ovirt'` (decision 7); downgrade leaves those rows empty, which is safe because a pre-3a daemon re-mints on the next `.vv`. Keep the ticket out of `ConsoleToken.export()` unless `include_secrets=True`, mirroring `CONSOLE_SECRET_FIELDS` (`db.py:246`) and how `get_console` takes `include_secrets`; check every caller of `export()` and of `get_token_by_token`/`get_tokens_by_console`/`get_token_by_session_id` still gets what it needs. Add to `db.py`: (1) `session_ticket(session_id, mint, lifetime, reuse_margin)` which, in one transaction, selects the token row by `session_id` `with_for_update()`, returns the stored `(ticket, expires)` if `expires - now >= reuse_margin`, and otherwise calls `mint(lifetime)` (a callable returning a `MintedTicket`) with the lock held, stores the result and commits; on an exception from `mint` it rolls back and re-raises; (2) `clear_expired_backend_tickets()`, nulling both columns where `backend_ticket_expires < now` or the token itself has expired, returning the count. Call (2) from the maintenance loop beside `_reap_expired_console_tokens` (`main.py:283-287`). Unit tests for the two functions in `test_db.py` on SQLite; note in a test docstring that SQLite ignores `FOR UPDATE`, which is why 3a.6 exists. The initial schema cannot be created on SQLite, so the migration is tested the way `test_timestamp_precision.py` tests `a8d3f6e1c9b2`: render it offline as MySQL and assert the SQL, both directions. Then run it up, down and up again against a MariaDB 11 container, with a static and an oVirt console present, and record in the commit message that the static ticket survived. Update `docs/schema.md` (the `consoletokens` table, and the `consoles.ticket` paragraph at `:149-171`, which must now say only static sources store a ticket there). Commit: "Give each session somewhere to keep its ticket."
3a.4	high	opus	none	The oVirt driver mints. In `kerbside/sources/ovirt.py`: set `MINTS_TICKETS = True`; give the CA fetch at `:46-48` a `timeout=MINT_TIMEOUT` (15 s, a module constant) and pass `timeout=MINT_TIMEOUT` to the SDK `Connection` wherever `_ensure_connection` builds one; change `get_console_for_vm(..., acquire_ticket=False)` to take `ticket_lifetime=None` and, when acquiring, call `console_service.ticket(ticket=OVIRT_SDK_TYPES.Ticket(expiry=int(lifetime)))` and return the ticket's value and the expiry the engine actually granted (the returned `Ticket` carries `.expiry`; read the ovirtsdk4 source in the lane's venv or `/srv/src-reference/ovirt/ovirt-engine-sdk` rather than guessing its units, and assert them in a test). Implement `mint_console_ticket(uuid, lifetime)` on top of it, returning `MintedTicket(value, now + granted_expiry)`. Update the direct handler (`api.py:380-479`) to use `sources.driver_class`, and to request `config.CONSOLE_TOKEN_DURATION * 60` seconds (decision 4); keep its existing secrets discipline (only the oVirt branch reads the source secrets). Tests in `test_sources_ovirt.py` and `test_api.py` (`:713-741` mock `get_console_for_vm` today; update them to the new signature and assert the lifetime requested). Commit: "Ask oVirt for a ticket that expires."
3a.5	xhigh	opus	none	Mint in the authorize path. In `kerbside/rpc/servicer.py:93-147`: after the token and console checks, look the source's driver up with `sources.driver_class(source['type'])`. If the class has `MINTS_TICKETS`, call `db.session_ticket(token['session_id'], mint, lifetime, REUSE_MARGIN)`, where `lifetime = max(token['expires'] - now, 0) + TICKET_SLACK` (decision 2; `TICKET_SLACK = 30`, `REUSE_MARGIN = 10`, module constants with a comment each saying why), and `mint` constructs the driver from `db.get_source(..., include_secrets=True)` and calls `mint_console_ticket(console['uuid'], lifetime)`, closing the driver in a `finally`. Audit per decision 6 (`db.add_audit_event`, as the function already does for "Channel created"); deny per decision 5 with `Denied(reason='console ticket unavailable')`. Put the minted ticket in `Target.ticket`; sources that do not mint keep `console['ticket']`. Never log the ticket, and check the existing `LOG.error('AuthorizeConnection failed: %s' % e)` at `:144` cannot carry one (an SDK exception can quote its request; log the class name there if in doubt, and say why in a comment). In `api.py:482-534` (`ConsolesProxyVirtViewer`), remove the oVirt mint and both `store_console_ticket` calls, so the proxied GET touches no ticket for any source type, and rewrite the comment block above it; then check whether `db.store_console_ticket` (`db.py:424`) has any caller left and delete it if not. Tests in `test_rpc.py`: a minting source mints once for three channels of one session; a stored ticket within the margin is reused; one past the margin is replaced; a mint failure denies, audits and stores nothing; a non-minting source still gets its row ticket. Commit: "Mint console tickets when a channel connects."
3a.6	high	opus	none	Prove the lock on MariaDB. SQLite ignores `FOR UPDATE`, so the unit tests cannot show that concurrent channels mint once. Add `kerbside/tests/functional/test_session_ticket_lock.py` (create the package if absent), skipped unless `KERBSIDE_TEST_SQL_URL` is set: create the schema on that URL, insert one token, and run eight threads calling `db.session_ticket` for the same session with a `mint` that sleeps 1 s and counts its calls. Assert exactly one mint and eight identical tickets. Add a second case where the stored ticket is inside the reuse margin, and assert one re-mint. The direct-qemu lane already runs MariaDB (`tools/direct-qemu/setup-mariadb.sh`); add one workflow step after it that runs this file against that database, calling a `tools/direct-qemu/run-ticket-lock-test.sh` rather than inlining commands. Check it fails with the `with_for_update()` removed, and record that in the commit message. Commit: "Test that one session mints once."
3a.7	high	opus	none	The oVirt lane proves the lifetime. In `tools/ovirt-e2e/drive-console.py`: correct the docstring's "expire in ~120s" (`:18-19`), and after step 4 hold the session until at least 30 s past the ticket's expiry, which is the token's remaining lifetime plus `TICKET_SLACK` from issue, then re-run the smoke client and assert the session is still up. Assert the console's `consoles.ticket` is empty for the lane's oVirt console, that the session's token row carries a `backend_ticket_expires`, and that exactly one `Minted backend console ticket` audit event exists for the session however many channels it opened. Never print the ticket. This lane runs in the merge tier only (`functional-tests.yml:380`), so run it once on the phase branch with a `workflow_dispatch` of `functional-tests.yml` selecting the oVirt target (`docs/testing.md`, the workflow table) before asking for merge, and record the run in Outcome. Commit: "Hold the oVirt session past its ticket."
3a.8	medium	sonnet	none	Docs. Rewrite oVirt's ticket lifecycle where survey finding 4 found it wrong: `docs/use-cases/ovirt.md:88-112` (flow step B moves from the `.vv` request to the first channel; the mermaid edge label at `:73` changes with it) and the limitations row at `:288` (kerbside now sets the lifetime: token lifetime plus 30 s proxied, `CONSOLE_TOKEN_DURATION` direct); `ARCHITECTURE.md:269-280` (the driver row, and the paragraph on where tickets are stored); `docs/console-sources.md` (the oVirt section, plus the two-sessions note from decision 8 and the supersession finding); `docs/proxy-architecture.md:95-105,203-235` (authorize now calls a source driver for minting sources, and can deny when the source is unreachable); `docs/configuration.md:18` (`CONSOLE_TOKEN_DURATION` now also bounds oVirt ticket lifetimes). Every claim about ticket lifetime must agree across these pages: grep `docs/ ARCHITECTURE.md tools/ovirt-e2e` for `ticket` near `minute`, `second`, `short-lived` and `expir` and reconcile each hit. Describe the code as it is: no references to this plan, its phases or its decisions. Commit: "Describe where console tickets come from now."

Risks and mitigations¶

A hung mint stalls every channel of a session. The row lock serialises them behind it. Mitigation: decision 5's timeouts. 3a.4 must show the SDK connection timeout actually applies to the ticket call, not only to login, by reading ovirtsdk4 and testing it with a stalled fake; the management session checks that test exists.
gRPC worker exhaustion. Each waiting channel holds a worker from API_GRPC_WORKERS for up to MINT_TIMEOUT. A session opens about six channels, so a few simultaneous session starts against a slow engine can fill the pool. Mitigation: record the worst case in docs/proxy-architecture.md (3a.8) and note in Future work that a per-session wait could move off the pool if it bites. Not engineered now, because the engine is fast when it is up and a slow engine already fails sessions.
A secret leaks through an exception. The authorize path's catch-all logs %s of the exception, and the SDK and requests can both quote a request. Mitigation: 3a.5 audits that path, and 3a.7's lane greps the daemon log for the ticket value, as drive-console.py already avoids printing it.
Rolling upgrades. A pre-3a daemon cannot authorize an oVirt channel once the migration has cleared the row (decision 7). Mitigation: the release note says to upgrade every node's daemon together; kerbside's migrations are already a shared, one-shot step, so this adds no new procedure.
The unit tests pass on SQLite and prove nothing about locking. Mitigation: 3a.6, on the real engine, with a mutation showing it fails without the lock.

Definition of done¶

Bugs fixed during this work¶

oVirt console tickets lived for the engine's default 7200 s. Kerbside now asks for the session's remaining token lifetime plus 30 s, and stores the expiry the engine echoes back.
One oVirt ticket was shared by every session on a console through the Console row, so a new .vv reset the password under an earlier session's feet. Each session now holds its own ticket on its consoletokens row.
The oVirt CA certificate fetch had no timeout. It is now bounded, and a failure marks the source errored and still removes its temporary file.
get_console_for_vm leaked an SDK connection on every API .vv. It now closes any connection it opens.
The drivers read a source's name from source, but database rows carry name, so a CA failure on an API path raised KeyError instead of erroring the source.
The direct .vv for a VM with no SPICE console was handed out with password=None. It now fails with 502.

Future work¶

Move waiting channels off the gRPC worker pool if a slow source ever exhausts it (see Risks).
Remember a failed mint for a few seconds. Against a hung engine each channel of a session takes its own MINT_DEADLINE turn behind the row lock, and the later ones exceed MariaDB's 50 s lock wait and fail as INTERNAL rather than as a denial. The session fails either way.
Serialise mints for one console across daemons. Within one daemon a new mint waits for a still-running worker on the same console, so an abandoned worker cannot replace a newer ticket; a worker abandoned on another node still can. The row lock covers the callers, not their abandoned workers.
Issue #319: the proxied .vv GET still mints a token.
/console/direct still hands a raw hypervisor ticket to its caller, though since #537 (issue #134) only to an administrator. Decision 4 bounds how long that ticket lives.

Execution notes¶

Implementation started 2026-10-25 (AEDT) on branch proxmox-source-phase-03a-minting, on develop at d4a4d49. The plan was merged in #506 but sat unimplemented for four weeks, so it was re-surveyed first. Twenty-one commits had touched its target files. The design held, and these premises moved:

2d543e7 keys consoles on (source, uuid) and points the consoletokens cascade at the pair. store_console_ticket now takes both. The migration's join on consoles.source is unaffected.
The initial schema cannot be created on SQLite, so migrations are tested as offline MySQL SQL plus a MariaDB run. 3a.3's brief and the definition of done said SQLite; both are corrected above.
3fcba92 (#533) moved every time.time() column to Double. ConsoleToken.expires is still an Integer of whole seconds, so backend_ticket_expires is an Integer too, matching the row it sits on. Decision 1 is otherwise unchanged.
537 closed issue #134: /console/direct is now¶
administrator-only. Future work is corrected above. Decision 4 still applies.
The merge tier runs by workflow_dispatch, not a bot retest. 3a.7's brief is corrected above.
oVirtSource.get_console_for_vm never closes an SDK connection it opened itself, so the API paths leak one per .vv. 3a.4's mint hook closes what it opens.
Every line reference in the step table has moved. The briefs given to the agents carry the current ones; the table is left as written.

Back brief answered 2026-10-25: decision 1 confirmed before 3a.3. The branch may be pushed and the direct-qemu and oVirt workflows dispatched on it without a pull request; the pull request waits for a separate ask. 3a.2's measurement runs in CI while 3a.3 and 3a.4 proceed, and 3a.5 waits for its result.

3a.2 ran the full rotation lane locally before CI, against qemu 10.0.13, MariaDB 11 and the Rust proxy: a connected session survived set_password and expire_password now, and a fresh session through the proxy presenting the old password was refused with PermissionDenied. That settles the back brief's second gate; 3a.5 may rely on it. The step added two checks beyond its brief and both were kept: a refusal check after set_password as well as after expire_password (once the password is expired qemu refuses everything, so only the first proves the old password is refused for being wrong), and a positive control connecting with the new password.

It also found that spice-server drops its connected client when a new client links. Decision 8's reasoning is therefore wrong for two sessions on one VM: the second session's main channel disconnects the first session outright, whatever the password, today and after this phase. Supersession still matters within one session, where a re-mint for a late channel leaves the session's established channels up, which is what 3a.5 relies on. 3a.8 documents the two-session case as a disconnect, not as a revoked password.

The back brief's third gate fired before 3a.4 ran. ovirtsdk4's Connection(timeout=) sets pycurl.TIMEOUT on the API handle (ovirtsdk4/__init__.py:413), so it bounds the ticket call, but the SSO login runs on its own curl handle with no timeout (_get_sso_response, :654-690), and a fresh connection logs in on its first request. Decision 5's timeouts therefore cannot bound a whole mint. The operator chose, on 2026-10-25, a deadline in the authorize path: 3a.5 runs the mint in a worker thread and waits at most a fixed deadline, denying on expiry and releasing the row lock, and a stuck worker's late result is discarded. It wraps whichever driver mints, so phase 4 inherits it. 3a.4 keeps the SDK and requests timeouts as the inner bound.

3a.4 went beyond its brief in three ways, all kept: the direct handler mints through mint_console_ticket, so a VM with no SPICE console now fails instead of handing out a .vv with password=None; a failed direct mint returns 502 and logs only the exception's class name; and a CA fetch that raises marks the source errored rather than raising from __init__, so its temporary CA file is still removed. It also closes the SDK connection that get_console_for_vm used to leak.

3a.5 put the deadline in kerbside/sources/minting.py (mint_with_deadline, MINT_DEADLINE = 20, a daemon worker that builds, uses and closes the driver), and on review the direct .vv path was moved onto it too, so a stalled login holds neither a gRPC worker nor a web worker. The mint now runs before the channel is recorded, so a denied channel writes no Channel created event, as an invalid-console denial already did not. db.store_console_ticket lost its last caller and was deleted. Review also found that the drivers read a source's name from source, as the scrape loop's sources.yaml entries carry it, while db.get_source() rows carry name, so a CA failure on any API path raised KeyError instead of marking the source errored. That predates this phase on the .vv paths; it is fixed where both minting paths now turn a row into driver arguments.

The first automated review of #604 found that a worker abandoned at the deadline still finishes its ticket request if the engine is slow rather than hung, replacing the hypervisor's password after a newer mint has stored its own. minting.py now keeps the running worker per console and makes a new mint wait for it, which also caps stuck workers at one per console; a worker stuck for five minutes stops blocking, so a connection that never answers cannot disable a console until restart. The rotation script now fails if the new password reaches an uploaded artifact.

Outcome¶

All eight steps landed on proxmox-source-phase-03a-minting on 2026-10-25 (AEDT), one commit each, after a re-survey commit. 536 unit tests pass.

The oVirt lane was dispatched on 581f5de, which carries every code and harness change in the phase (only 3a.8's docs follow it), and passed: run 40148744908. In it one session minted one 8-character ticket for four channels, expiring 30 s after its token, with no failed mint. The session was held 30 s past that expiry, re-ran the smoke client with all four channels still open, and the maintenance loop cleared the expired ticket. The ticket appeared in none of the 8 collected files nor the audit trail. The migration cleared the stored ticket of 0 oVirt consoles on that fresh install.

The direct-qemu workflow has no workflow_dispatch trigger, so the MariaDB lock test and verify-ticket-rotation.sh first ran in CI on #604, at ed59e13, and passed: run 40158168216. Both lock tests passed (one mint for eight concurrent channels, with an empty row and inside the reuse margin). The rotation lane kept the same five channels up through set_password and expire_password, refused a fresh session presenting the old password after each, and accepted the new password in the positive control, with the new password in no uploaded artifact.

Back brief¶

Before executing any step of this plan, back brief the operator on the intended approach and on any deviation from this plan found during implementation. In particular:

confirm decision 1 before 3a.3 writes the migration. It reverses the master plan's in-memory design, and a schema change is the expensive thing to redo;
report 3a.2's measurement before 3a.5 relies on it. If a connected client does not survive a new password, stop: decision 3's re-mint path is unsafe, and Proxmox (phase 4) inherits the problem;
if 3a.4 finds the SDK cannot bound the ticket call with a timeout, say so before 3a.5 puts it under a row lock.

📝 Report an issue with this page