feat: declarative per-game provisioning pipeline (opt-in, WIP)#1610
Closed
moukrea wants to merge 16 commits into
Closed
feat: declarative per-game provisioning pipeline (opt-in, WIP)#1610moukrea wants to merge 16 commits into
moukrea wants to merge 16 commits into
Conversation
Phase 0 of a compatibility + code-quality effort: a declarative, launch-time per-game provisioning pipeline (pinned components + env + DLL overrides + registry patches + winetricks-style dependency verbs), resolved with precedence and device branching, applied idempotently and transactionally, and offline-safe. The design maps the existing architecture (Container/ImageFs, the manifest.json component catalog, the BestConfigService 3-stage server flow, GameFixesRegistry, PreInstallSteps/LaunchDependencies) and shows where the new pipeline hooks in behind an opt-in feature flag with zero regression for existing containers. Includes the recipe schema, the PrefixState-based testable engine + verb registry, the launch resolver, catalog integration, the migration plan for the 35 hard-coded fixes, and the machine-checkable oracle.
Phase 1 of the per-game provisioning pipeline. Adds the declarative, versioned recipe schema (kotlinx.serialization) covering identity/matching, component pins (referencing manifest.json ids), env, DLL overrides, registry patches, dependency verbs, file writes, cleanups, launch args, per-device overrides and provenance; a forward-compatible JSON codec; and a pure, Android-free validator (schema version, DLL-override tokens incl. the repo's 'd'/'disabled' forms, dword parsing, device-condition predicates, file exclusivity). Tested headlessly with JUnit4 (no Robolectric): 4 serialization + 8 validation tests over valid/invalid fixtures. All new code passes ktlint.
… registry Phase 2. Adds: - PrefixState: a narrow, mockable abstraction over the prefix + container (env, component pins, registry, drive_c files, launch args, markers), with a production FilePrefixState (Container/WineRegistryEditor/EnvVars) and an in-memory test implementation for fully headless testing. - ProvisioningEngine: applies a recipe with device-overlay merging, content-hash idempotency (re-apply on unchanged prefix is a no-op), and transactional rollback on failure. Dependency verbs are applied leniently and never fail-closed: a verb that cannot install (e.g. offline) leaves the recipe unmarked so the next launch self-heals. - Verb registry: a winetricks-style, data-driven verb executor (place DLLs, set overrides, write registry, or emit a guest installer command) seeded with priority verbs (vcrun2015-2022, dotnet48, openal, d3dcompiler_47, d3dx9_43). Definitions reference official vendor downloads only; nothing is bundled. Headless tests (23): idempotence, device branching, transactional rollback, offline self-heal, DLL-override merge, verb placement/idempotence/commands, and a conformance harness applying a full recipe to an in-memory prefix.
Phase 5. Translates all 31 entries of the hardcoded GameFixesRegistry (Steam, GOG, Epic) into declarative recipes shipped as a bundled, diffable catalog (migrated-fixes.json), loaded by MigratedFixCatalog. Each fix type maps to a recipe primitive: RegistryKeyFix -> registry (with <InstallPath> placeholder preserved), LaunchArgFix -> launch.args, WineEnvVarFix -> env/dllOverrides, DeleteFolderFix -> cleanup, PrefixFileFix -> files, IniFileFix -> iniPatches, GOGDependencyFix -> dependencies, CompositeGameFix -> a single merged recipe. Adds game-install-dir INI patching (iniPatches) to the schema and engine to cover IniFileFix losslessly. A parameterized conformance test validates, round-trips and applies every migrated recipe to an in-memory prefix and asserts it reproduces the original fix's effects (per-fix assertions). The legacy GameFixesRegistry remains the fallback while the feature flag is off.
Phase 3. Adds RecipeResolver, which selects a game's recipe by source precedence (user override > community catalog > built-in migrated catalog) and caches the result so a later launch is offline-safe: if every source is unavailable, the last resolved recipe is reused, and the resolver never fails closed (returns null only when nothing exists, where the caller falls back to the legacy path). Adds the built-in MigratedFixCatalogSource and a container-backed PrefixRecipeCache. Gated by a new opt-in PrefManager flag enablePerGameProvisioning (default off), so existing containers and the legacy GameFixesRegistry path are unaffected. Tests: source precedence, fall-through, offline fallback, and built-in catalog lookup.
…ng verbs Adds docs/device-validation-protocol.md (the human hardware-boot handoff: witness games incl. Mirror's Edge, candidate recipes, and the on-device A/B procedure), and a THIRD_PARTY_NOTICES section clarifying that dependency verbs only reference official vendor downloads at runtime (nothing bundled) and that verb/recipe facts are reimplemented from winetricks as an oracle (clean-room, no code copied).
…settings toggle Makes the opt-in pipeline actually run on-device: - XServerScreen: when enablePerGameProvisioning is on, resolve and apply the per-game recipe at the existing pre-launch fix point (falls back to the legacy GameFixesRegistry when no recipe matches); unchanged when off. - PerGameProvisioning.applyAtLaunch: resolves via user-override > built-in migrated catalog (cached for offline relaunch) and applies the declarative state synchronously (no downloads on the launch path). - ProvisioningEngine: split out a synchronous applyDeclarative() (no verbs) from the suspend apply(); behaviour unchanged (all unit tests still green). - UserRecipeSource: reads recipe overrides from the private and external files dirs (adb push-friendly) so any game can be targeted without a rebuild. - Settings > Info: an experimental 'Per-game provisioning' switch for the flag. Dependency-verb installation (downloads/guest installers) is intentionally not on this synchronous launch path yet; that is a follow-up.
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…apply - Apply semantics: passive launch apply is now set-if-absent everywhere (never overwrites a user's existing env/DLL/registry/launch/component values); an explicit force-apply overwrites and bypasses the idempotence marker. - GameHubCatalog: 19 per-game recipes derived from the open GameHub Lite / BannerHub catalog (config-file overlays for Sifu, Metro Exodus, Dying Light, GTA V, Sekiro, Skyrim SE, Bannerlord, Tiny Tina's, Kena, It Takes Two, Silent Hill f, ...) with web-verified Steam appids; paths normalized to the GameNative prefix user (xuser); junk (crash logs/caches/stubs) filtered out. - Resolver precedence: user override > GameHub catalog > migrated fixes. Facts only; no GameHub code or proprietary binaries (see THIRD_PARTY_NOTICES). Conformance test validates + applies all GameHub recipes headlessly.
… catalog + baseline) Reverse-engineered (clean-room) what makes GameHub boot games a vanilla prefix can't, and ported it: - gamehub-verbs.json: the full winetricks-seeded dependency verb catalog (99 verbs: vcrun2005-2022, d3dx9, d3dcompiler, dotnet20-48, physx, xact, xna, openal, gdiplus, dmusic/directshow, ...) with real download URLs, SHA-256, guest install commands and DLL overrides. Loaded into the verb registry. - gamehub-baseline.json: GameHub's default-prefix provisioning — the common Windows runtimes + Box64/FEX tuning (BOX64_DYNAREC_* / FEXCORE_*) — applied under every game at launch (the reason UE3/PhysX games like Mirror's Edge boot under GameHub via the default path, no per-game entry needed). - Full 575-component GameHub catalog mapped; resolver precedence user > GameHub per-game > migrated; baseline applied beneath all. - docs/gamehub-compat-mechanism.md documents the mechanism + the port + the on-device validation handoff. THIRD_PARTY_NOTICES attributes GameHub Lite / BannerHub + winetricks (facts only; no GameHub code or binaries). Tests: verb catalog loads, baseline validates/applies tuning, all baseline deps resolve in the registry.
This is the boot-maker: GameHub bakes the common runtimes into its imagefs; GameNative now provisions them at launch instead. - ProvisioningDepsStep (a PreInstallStep): resolves the baseline + per-game installer verbs (VC++ 2005-2022, PhysX, .NET 4.8, XNA), downloads each official redistributable via SteamService.fetchFile (SHA-256 verified) into a shared per-prefix staging dir C:\.gnprov\<verb>\, and runs them silently in the Wine guest through the existing pre-install chain (markers, wineserver -k, splash). Reuses the same mechanism that installs a game's bundled redists. - ProvisioningInstallers: pure, unit-tested command builder (silent flags per verb; .msi via msiexec, .exe direct). - Opt-in (PrefManager.enablePerGameProvisioning); zero behavior change when off. Hardening from an adversarial review pass: - Withhold the completion marker when any verb fails to stage, so a transient download failure retries next launch instead of leaving the prefix permanently half-provisioned; add PROVISIONING_DEPS_INSTALLED to the 'Verify Files' reset list for manual re-provision. - PhysX silent switch /s (NSIS), not /quiet /norestart. - Drop dotnet40/45/46 from auto-run (wrong flags, fragile on Wine). - Stop BuiltinVerbs from shadowing the SHA-256-pinned catalog entries for vcrun*/dotnet48 (keep only the DLL-placement value-adds); installer verbs now resolve to the pinned official downloads. - Guard the pre-download verify against unreadable files; bound each download with a 180s timeout so a stuck CDN can't wedge launch.
…d re-apply UI
The provisioning JSON (baseline/recipes/verbs/migrated) was loaded via
ClassLoader.getResourceAsStream from src/main/resources. That passes in JVM unit
tests but returns null inside the installed APK, so on-device every recipe/verb
loaded as null: no baseline, no resolved deps, no pre-install command, no
'Installing prerequisites' splash — the entire feature silently no-op'd. Every
other bundled JSON in the app loads via context.assets.open(); this now does too.
- Move provisioning JSON to src/main/assets/provisioning/; new ProvisioningAssets
loads via the app context (set at startup in PluviaApp), with a classpath
fallback so pure-JVM unit tests still resolve them (test sourceSet adds
src/main/assets to its resources).
- All four loaders (GameHubBaseline, GameHubCatalog, MigratedFixCatalog,
GameHubVerbCatalog) go through ProvisioningAssets.
UI so the user can trigger and confirm provisioning (was zero UI):
- 'Re-apply provisioning' per-game menu option (shown only when the flag is on).
It is the explicit force action: force-applies baseline + per-game recipe now
(overwriting prior provisioning), clears the deps marker so the Windows-runtime
installers re-run next launch, and shows a snackbar summary ('baseline applied
· recipe X · N runtimes install on next launch: vcrun2010, physx, ...').
- PreInstallSteps.clearProvisioningDepsMarker() helper.
Both flavors' provisioning unit tests stay green.
Mirror's Edge & co. fail with 'Application load error 3:0000065432' — a Steam DRM
failure (no Steam client to validate ownership), NOT a missing runtime. GameNative
already bundles the open-source fix (Goldberg steam_api emulator, cold-client
steamclient loader, Steamless) and picks a path from the per-container
useLegacyDRM/unpackFiles/launchRealSteam toggles — but it never selects one per
game, so a fresh Steam-DRM title defaults to the cold-client path and the stub
survives.
This adds the missing appid-keying declaratively:
- GameRecipe gains an optional steamDrm { strategy, unpack } block
(SteamDrmStrategy: auto | legacy_goldberg | cold_client | real_steam).
- PerGameProvisioning.applySteamDrm() maps it onto the container's existing DRM
toggles. Set once per container (respects later manual changes); forced on the
explicit 'Re-apply provisioning' action (and surfaced in its snackbar).
- Mirror's Edge recipe (Steam 17410) selects legacy_goldberg + unpack — the
investigation's leading hypothesis for its 2008 SteamStub; needs on-device
confirmation.
Selects only already-shipped mechanisms; bundles nothing new and copies no
GameHub code (mirrors only the semantics of its per-game fakeSteamClient flag).
Conformance test updated: a GameHub recipe may carry config files OR a DRM
directive.
…ctric asset proof
- ProvisioningDepsStep now emits per-verb progress to the boot splash
('Preparing dependencies: vcrun2010 (1/11)…') while it downloads. The download
is synchronous and can be 100-200 MB, so without this a first launch looked
frozen with no indication anything was happening — the exact 'I see nothing'
symptom reported.
- ProvisioningAssetsAndroidTest (Robolectric): opens the provisioning JSON via the
real Android AssetManager and through ProvisioningAssets, proving the on-device
load path works (the pure-JVM tests only exercised the classpath fallback that
masked the original null-on-device bug). isIncludeAndroidResources=true makes
Robolectric read the merged assets exactly as the installed APK does.
…he launch chain ProvisioningDepsStepWiringTest (Robolectric): asserts ProvisioningDepsStep is in PreInstallSteps.registeredStepsForTest() (so it runs at launch) and that appliesTo is true iff (flag on AND not yet provisioned) — inert when the flag is off, idempotent once the marker exists. Adds registeredStepsForTest() accessor. Together with ProvisioningAssetsAndroidTest (data loads via the real Android AssetManager) and ProvisioningInstallersTest (install command builds), the full launch chain is now proven in CI; only the network download + in-guest install remain device-only.
…d-truth check) Answers the user's 'on doit pouvoir en etre sur' — a way to verify provisioning actually happened, independent of catching a launch-time splash. New per-game menu option (flag-gated) shows a snackbar reading real on-disk state: - resolved recipe, runtimes actually downloaded (N/M, by scanning drive_c/.gnprov/), whether the install marker is set, and the active DRM mode. PerGameProvisioning.statusSummary() + PreInstallSteps.isProvisioningMarked().
…and with live progress
Makes verification possible WITHOUT launching a game (the user's 'on doit pouvoir
en etre sur'): reapplyNow now prefetches the resolved runtimes into the shared
drive_c/.gnprov staging dir, reporting per-runtime progress to the snackbar
('Downloading runtime 3/11: vcrun2010...' -> 'Runtimes downloaded: 11/11'). The
user can trigger it from the menu and watch it work.
Purely additive: ProvisioningDepsStep (the launch-critical installer) is unchanged
and simply finds the files already staged (skips re-download); integrity is still
(re)verified there at launch. reapplyNow is now suspend (runs on the existing IO
coroutine from the menu).
Author
|
Closing for now at the author's request — this is still WIP and being validated on-device. We'll reopen a single, complete PR with all the work once it's ready. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Draft / WIP. A declarative, opt-in per-game provisioning pipeline that closes a class of
"won't even launch" compatibility gaps. It makes the Wine-prefix state a game needs — pinned
components, environment variables, DLL overrides, registry patches, and winetricks-style
dependency verbs — a single declarative, shareable recipe that is resolved by precedence,
branched per device, and applied idempotently and transactionally at launch.
Design rationale and architecture:
docs/provisioning-pipeline-design.md.Why this fits the project scope
the box without manual tweaks" and "reducing complexity and duplication"). It extends the existing
"auto-applied known configs" mechanism (
BestConfigService) and migrates all 31 hard-codedGameFixesRegistryfixes out of Kotlin into declarative, diffable recipes.PrefManager.enablePerGameProvisioningflag(default off). With the flag off, behaviour is byte-for-byte the current path and the legacy
GameFixesRegistryremains the fallback.What's in this PR
PrefixState, so the wholeengine is unit-testable headlessly (no device, no real Wine).
vcrun2015-2022,dotnet48,openal,d3dcompiler_47,d3dx9_43, …). Verbs reference official vendor downloads only — nothing isbundled or redistributed; facts are reimplemented from winetricks (clean-room).
offline-safe caching — it never fails closed.
GameFixesRegistryfixes into a bundled recipe catalog, each covered by aconformance test.
Recording
No UI/runtime change in this draft (the path is behind an off-by-default flag and the on-device
integration is still WIP — see below). On-device validation is a human handoff documented in
docs/device-validation-protocol.md.Type of Change
Checklist
#code-changes, I have discussed this change there. (Opening as draftto gather direction first, per
CONTRIBUTING.md.)opt-in behind a flag with zero regression when off.
CONTRIBUTING.md.Not in this PR (tracked WIP / follow-up)
LaunchDependency) + a real downloadVerbContext.Testing
Headless unit tests run under the existing CI tasks
:app:testLegacyDebugUnitTestand:app:testModernDebugUnitTest: schema validity, engine idempotence + transactional rollback,offline self-heal, device branching, resolver precedence/offline fallback, verb
placement/idempotence/commands, and per-migrated-fix conformance for all 31 fixes.
Summary by cubic
Adds an opt-in, declarative per-game provisioning pipeline with baseline Windows runtime installs, per-game Steam DRM selection, and a read-only “Provisioning status” menu to verify on‑disk results. “Re‑apply provisioning” now pre‑downloads required runtimes with live progress so users can confirm setup without launching a game.
New Features
kotlinx.serialization).ProvisioningEngineoverPrefixState; launch does set‑if‑absent apply (force‑apply available).C:\.gnprov\<verb>\, verifies SHA‑256, 180s timeouts; runs silently; withholdsMarker.PROVISIONING_DEPS_INSTALLEDon failure so it retries and is covered by “Verify Files”; boot‑splash shows per‑verb progress.C:\.gnprov\with live per‑runtime progress; next launch reuses the staged files (integrity re‑checked).PrefManager.enablePerGameProvisioning(default off) + Settings toggle, per‑game “Re‑apply provisioning,” and read‑only “Provisioning status” (shows resolved recipe, downloaded runtimes N/M, install marker, and active DRM mode).GameFixesRegistryentries moved toapp/src/main/assets/provisioning/migrated-fixes.json, each covered by tests.Bug Fixes
ProvisioningAssets(classpath loading returned null in APK); unit tests includesrc/main/assetson the test classpath; RobolectricProvisioningAssetsAndroidTestproves the AssetManager load path.ProvisioningDepsStepis registered in the launch chain and only activates when the flag is on and the deps marker is absent.Written for commit 81e04a8. Summary will update on new commits.