Skip to content

Scripted Windows test guest on KVM/QEMU for driver runtime tests - #64

Closed
Chuccle wants to merge 18 commits into
masterfrom
claude/windows-guest-test-rig-vl5f6l
Closed

Chuccle wants to merge 18 commits into
masterfrom
claude/windows-guest-test-rig-vl5f6l

Conversation

@Chuccle

@Chuccle Chuccle commented Oct 3, 2026 •

Copy link
Copy Markdown
Owner

Before: the driver only ever met a real Windows kernel on a developer's VMware VM, driven by hand through vmrun. CI built and unit-tested it but never loaded it.

After: any Linux host with /dev/kvm (a stock GitHub ubuntu-latest runner, a cloud VM with nested virtualization, a Linux dev box) can boot a scripted Windows guest, install the packaged driver in it, test it against the package's Linux server-rs on the host, and return a pass/fail verdict with logs and crash dumps. CI does that as a fixed test for every successful build, and anyone can work inside the guest by hand through one CLI.

The input is the blorg-package-windows-x64 artifact from #63, so the rig tests exactly what ships and builds nothing itself.

How

  • ci/guest/image/build-image.sh: unattended Windows Server 2022 Server Core install from Microsoft's evaluation ISO, every input pinned by SHA-256. Setup-GoldenImage.ps1 installs OpenSSH (key-only, PowerShell shell) and the QEMU guest agent, turns test signing on and boot recovery off, sets Driver Verifier on BlorgFS.sys, keeps kernel dumps, and disables Windows Update. The result is booted once and checked over SSH before it is published.
  • ci/guest/host/guestctl: up, down, ssh, ps, push/pull, reboot, snapshot/revert, screenshot, qga-exec, qmp. SSH is forwarded on host loopback only.
  • ci/guest/run-guest-tests.sh: one end-to-end run. server-rs runs on the host and the guest reaches it at 10.0.2.2; a deterministic corpus comes from host/make-corpus.py. A bugcheck or unexplained reboot fails the run even if every step passed. The verdict is also a blorg guest-verdict check annotation.
  • .github/workflows/guest-runtime.yml: a fixed test that takes no commands. Runs on workflow_run of Build BlorgFS, on PRs touching the rig, and on demand. Image cached by recipe hash and quarter; runs queue rather than cancel; no secrets.
  • .github/workflows/guest-session.yml: manual only, gates nothing. Runs a script of guestctl commands against a live guest, for working in it without a KVM host of your own.
  • ci/guest/README.md: how to run all of it by hand.
  • ci/guest/infra/azure/main.bicep: optional long-lived KVM host.

Temporary: rig PR runs take #63's package build until #63 is merged; that commit comes out afterwards.

The driver only ever met a real kernel on a developer's VMware VM. This
adds a Windows guest that any Linux host with /dev/kvm can run unattended,
and a workflow that loads every built package into it.

- ci/guest/image: builds a Server Core golden image from the evaluation
  ISO, with OpenSSH, the QEMU guest agent, test signing on and boot
  recovery off, and checks it over SSH before publishing it.
- ci/guest/host/guestctl: one CLI to boot, ssh, push/pull, reboot,
  snapshot/revert, screenshot and guest-agent-exec the guest, for CI and
  for agents acting in the guest.
- ci/guest/in-guest: prepare (verifier, dumps), run (package integrity,
  corpus, server-rs, install, listing, Test-BlorgCorrectness, extra
  suites), and collect diagnostics with a bugcheck verdict.
- ci/guest/run-guest-tests.sh: one end-to-end run with an exit code.
- guest-runtime.yml: runs it on ubuntu-latest after each successful build,
  against the blorg-package-windows-x64 artifact.
- ci/guest/infra/azure: optional long-lived nested-virtualization host.
Copies the whole tests/guest-suites directory (only its top-level *.ps1
run), and third_party/schemas/conformance into suites/contract, where the
Contract suite from the API alignment work looks for it. The workflow
fetches the schemas submodule without credentials, since it runs PR code.
…r's image cache

server-rs now runs on the KVM host as the Linux build and the guest reaches
it at 10.0.2.2, so the driver's traffic crosses a real NIC and the backend is
case-sensitive, as deployed. The corpus is generated on the host by
make-corpus.py (one generator for both topologies); suites' fixtures ship as
suites/<name>.corpus/. The in-guest Windows server remains as a fallback when
the package has no Linux binary.

A workflow_run triggered by a pull request runs in master's cache scope, so
it now uses the default branch's rig scripts and never saves the golden
image.
guest-runtime.yml takes a session input: bash run against the live guest
after the tests, with guestctl on PATH, its transcript and files returned in
the guest-results artifact. PRs touching the rig run session-probe.sh, which
exercises every channel once (ssh, push/pull, the driver's volume, screenshot,
guest agent, snapshot/revert).
…r into the image

The package now carries server/linux-x64/server-rs, so the in-guest Windows
server fallback is gone along with its corpus push and -CorpusDir. Driver
Verifier is set in the golden image, so a default run needs no reboot before
its tests, and the image is compressed with zstd for faster first reads.
@Chuccle
Chuccle force-pushed the claude/windows-guest-test-rig-vl5f6l branch from 5387da9 to a044fb3 Compare October 3, 2026 22:12
The first image build ran 67 minutes with no way to see why from outside the
artifact. Changed screens now go to the log as base64 every two minutes, as
do a session's screenshots, so an agent that can read logs but not download
artifacts can still see the guest.
Agents behind an egress proxy can read the checks API but not job logs
or artifacts; the verdict (blorg guest-verdict) and the agent session's
transcript tail (blorg guest-session) now reach them as annotations.
guest-runtime.yml no longer takes commands: the package from build.yml,
an image from pinned inputs and the suites in the commit, nothing else.
Acting in the guest from a remote session moves to guest-session.yml,
manual only and gating nothing. The host setup both share is one
composite action, ci/guest/action.yml.

Image inputs (install ISO, virtio-win, OpenSSH zip) are pinned by
SHA-256; an unpinned one is reported with its hash so it can be pinned.
Runs queue instead of cancelling, so an image build is never thrown
away, and the image builds even when no package is found.
…host

The guest section moves out of AGENTS.md into ci/guest/README.md, written
for a person running the scripts on any Linux host with KVM. Comments,
the workflows and the session script no longer assume who is driving.
A '--' inside an XML comment made autounattend.xml malformed, so Setup
ignored it and sat at the language screen until the build timed out.
build-image.sh now checks the answer file is well-formed before booting.
- Install-BlorgFS.ps1 gets its .inf and .cer paths explicitly: under
  powershell -File, Windows PowerShell 5.1 leaves $PSScriptRoot empty in
  param() defaults.
- A child script's stderr no longer aborts the step under 'Stop'; its
  exit code decides.
- A step that returned $true was recorded as a skip ($true -eq 'skip' is
  true in PowerShell).
- The guestctl probe runs even when the tests fail.
Versioned URLs and SHA-256 pins for the Windows evaluation ISO and
virtio-win 0.1.302, and a SHA-256 pin for the Win32-OpenSSH zip, taken
from the first full image build.
The first full run installed the driver on Server 2022 (build 20348) and
sc start failed with 127, procedure not found: BlorgFS.vcxproj targets
NTDDI 0x0A000010 (24H2, build 26100), so it imports kernel routines 2022
does not have. The golden image is now Server 2025 evaluation (26100).
Its ISO hash is pinned after this build reports it.
The probe failed on 'sc query BlorgFS' after the driver bugchecked the
guest. Whether the driver is healthy is the tests' verdict; the probe
only checks that every way into the guest works.
When a guest run brings minidumps back, a windows-latest job (which has
cdb) runs tools/Get-CrashVerdict.ps1 over them with the driver's PDB from
the same Build BlorgFS run, and reports the bugcheck, faulting line and
stack in its log and as a 'blorg guest-crash' check annotation.
Show-CrashAnalysis.ps1 does the same by hand on any Windows machine.
Get-CrashVerdict.ps1 gains -SymbolDir for the driver's own PDB.
A minidump's !analyze reports BUGCHECK_CODE rather than the full dump's
'BugCheck' banner, so the verdict left Bugcheck empty for the guest's
0xD1.

Chuccle commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner Author

Windows guest (KVM) is red because of a driver bug, not because of this PR.

The package under test is #63's build (run 37157505659). Its driver bugchecks with 0xD1 in DriverEntry, at Driver.c:1112. The cause is BlorgPrewarmSocketPool, which reads ai_addr while holding SocketPool.Lock at DISPATCH_LEVEL. That address comes from WskGetAddressInfo and lives in paged pool, so Driver Verifier makes the crash happen on every run. The same code is on master. The rig itself works: it detects the crash, the crash-analysis job gets the stack, and the channel probe passes.

A fix exists as a patch that moves the address copy before the spinlock. It is not ported into this PR, because this PR does not build the package. The fix has to land where the package is built (#63, or the consolidated #68), and this check will turn green once the guest tests a package that includes it.

Chuccle commented Oct 4, 2026 •

Copy link
Copy Markdown
Owner Author

Replaced by #70, which carries the Windows test guest (now Windows Server 2025, Release driver under Driver Verifier) and the driver fixes it found are in #69.

@Chuccle Chuccle closed this Oct 4, 2026
@Chuccle
Chuccle deleted the claude/windows-guest-test-rig-vl5f6l branch October 4, 2026 17:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant