WIP: ENH: igenerator pkl correctness fixes + CastXML content-addressed cache (supersedes #6486)

[hjmjohnson]

Cuts Python-wrapping CI build time by ~31 min per run on 2-core Azure agents (fleet median 175 min → 144 min, −18%; best same-code cold→warm pair −20%) and by up to −64% on GHA ubuntu-24.04 combined with warm ccache — via two latent pkl-tracking correctness fixes plus a two-level content-addressed CastXML cache. Populating the cache is free (cold runs match the fleet median). Validated by a 49-test cross-platform pytest suite wired in as a WrappingInfrastructure ctest. Supersedes #6486.

Measured CI performance — this PR's own builds (AzDO ITK.Linux.Python, 2-core)

Fleet baseline: the 32 most recent standard builds on this pipeline (other PRs + main, warm ccache, no CastXML cache) have median 174.9 min (range 158.7–235.8).

Build class	Builds	Duration
Standard (fleet median, n=32)	—	174.9 min
This PR, cache cold / populating	16147, 16173	177.1, 179.0 min
This PR, cache warm	16158, 16160, 16181	143.8, 145.0, 142.5 min

• Steady-state saving: ~31 min (−18%) vs fleet median; warm runs are also far more consistent (2.5 min spread vs 77 min fleet spread).
• Best controlled measurement: builds 16173 → 16181, identical source one day apart: 179m00s → 142m29s = −36m31s (−20%).
• Cache-population overhead is free — cold runs land at the fleet median.
• Build 16181 breakdown: configure 51s / build 131m / test 6m; ccache hit rate 99.97% — the CastXML cache removes the wrapping-generation time that ccache cannot touch.

On 4-core GHA ARM runners the total is flat (±1 min) as expected: with enough cores CastXML is not on the critical path. The cache targets core-count-limited runners.

Measured CI performance — prototype runs (PR #6486, GHA + AzDO)

GHA ubuntu-24.04 (2-core, combined CastXML cache + ccache cold→warm):

Phase	Cold (2026-06-28)	Warm (2026-06-29)	Δ
CastXML (816 jobs)	31m 45s	23 sec	−98%
Total	5h 1m 52s	1h 49m 50s	−64%

AzDO ITK.Linux.Python — CastXML cache isolated (ccache warm on both sides):

	Cold (16132)	Warm (16133)	Δ
Build and test	167m 54s	132m 15s	−35m 39s
Total	2h 52m 46s	2h 20m 23s	−32m 23s (−19%)

72-core local workstation (NVMe, ccache warm): warm speedup ~4% — CastXML parallelizes off the critical path; the cache is for CI-class machines.

What this fixes (correctness)

Ninja graph gap — igenerator.py writes one .pkl file per wrapped
class (~816 files per full ITK build) into itk-pkl/ as side effects.
None of those files appear as CMake OUTPUTs; only the .index.txt
manifests do. When itk-pkl/*.pkl files are absent (fresh clone, partial
ninja -t clean, CI agent reusing a workspace with a stale build tree)
while the .index.txt files survive, ninja considers igenerator.py
up-to-date. pyi_generator.py then fails with:

No pickle files were found in the pkl directory

Fix (commit 1): a per-module <Module>.stamp file is written by
igenerator.py after all pkl work for that module completes and is
declared as a CMake OUTPUT. Ninja now reruns igenerator.py whenever
the stamp is absent, guaranteeing the pkl files exist before
pyi_generator.py runs.

Parallel write race — the ~96 concurrent igenerator.py jobs (one per
ITK module) all write individual .pkl files into the same itk-pkl/
directory with no locking. Under adverse scheduling this produces torn
reads or silently truncated files.

Fix (commit 2): replace the directory of per-class .pkl files with a
single WAL-mode SQLite database. SQLite WAL mode is designed for
concurrent writers; the race is structurally eliminated. The .index.txt
manifests now hold DB keys (not file paths), and pyi_generator.py
queries the DB via those keys, self-healing if indexed keys are missing.

What this adds (performance)

Commit 3 defers import pygccxml until after the submodule list is
built from the .mdx files, separating the pure-I/O discovery phase from
the pygccxml/CastXML-dependent processing phase.

Commit 4 adds itk-castxml-cache.py, a transparent wrapper for the
CastXML binary with a two-level content-addressed cache:

• L1 (no subprocess, ~0.1 s): SHA-256 of CastXML binary content +
  .castxml.inc + .cxx content + flags → manifest with L2 key, a
  ccache-style dependency list, and an include-dir fingerprint that
  detects header shadowing.
• L2 (preprocessor pass, ~0.75 s): SHA-256 of castxml -E output with
  line markers stripped → output.xml.gz. Path-independent: any build
  directory on the same source tree produces the same L2 key.

Concurrency-safe (atomic staging + rename), size-capped with synchronous
eviction (ITK_WRAP_CACHE_MAX_SIZE, default 2 GB). Enable via
ITK_WRAP_CASTXML_CACHE (default ON); cache root override via
ITK_WRAP_CACHE (default ~/.cache/itk-wrap).

CI wiring: Cache@2 tasks in all three Azure DevOps Python pipelines and
GHA arm.yml; configure-python-ci / build-python-ci /
test-python-ci pixi tasks.

Validation suite (commit 6)

Wrapping/Generators/Tests/: 49 pytest tests covering cache key
composition, dependency invalidation, header shadowing, eviction,
concurrency (parallel writers), pkl DB integrity, pyi_generator
pruning/self-heal, and tar-relocation of the cache between build trees
(models actions/cache / Cache@2 restore). Runs in ~7 s using an
instrumented fake CastXML with exact subprocess-count assertions
(cold=2, L2-hit=1, L1-hit=0).

Registered as ctest WrappingInfrastructure when pytest is available
(graceful skip otherwise); pytest added to the four Python CI package
installs. The same suite runs standalone on
{ubuntu-24.04, macos-14, windows-2022} × {Python 3.11, 3.13} — all six
legs green.

Why this supersedes #6486

PR #6486 (ci/linux-azure-disk-management) was a work-in-progress branch
used to accumulate CI benchmark data. It contains temporary benchmarking
artifacts, a vestigial igenerator LRU cache that was added then removed,
benchmark key-version bumps, and two files already merged separately
(#6530, #6531). This PR is a clean rewrite from upstream/main with the
same final implementation in six independently-reviewable commits.

[greptile-apps]

Greptile Summary

This PR fixes two latent correctness bugs in the Python wrapping pipeline (Ninja graph gap via a new per-module stamp file, and a parallel-write race via SQLite WAL-mode), then adds a two-level content-addressed CastXML cache (itk-castxml-cache.py) that delivers −19 % to −64 % CI build time savings. CI wiring is added for Azure DevOps (Linux, macOS, Windows) and GitHub Actions (arm).

• Stamp file fix: igenerator.py now writes a <Module>.stamp file declared as a CMake OUTPUT; ninja will re-run igenerator.py whenever the stamp is absent, guaranteeing pkl data exists before pyi_generator.py runs.
• SQLite pkl store: Per-class .pkl files in itk-pkl/ are replaced with a single WAL-mode SQLite database (itk-pkl-v1.db); .index.txt manifests now carry DB keys instead of file paths, eliminating concurrent-write races across ~96 parallel igenerator jobs.
• CastXML cache: itk-castxml-cache.py wraps the CastXML binary with an L1 (no-subprocess, ~0.2 s) / L2 (preprocessor pass, ~0.75 s) content-addressed two-level cache enabled by ITK_WRAP_CASTXML_CACHE=ON.

Confidence Score: 4/5

The correctness fixes (stamp file, SQLite WAL store) are well-designed and the cache implementation is solid; the only concerns are quality-of-life gaps that don't affect build correctness in the tested configurations.

The stamp-file and SQLite migration are structurally sound and CI-tested at scale. Three things warrant a second look: the default 5-second SQLite busy timeout in _open_pkl_db could produce OperationalError: database is locked under peak 96-writer concurrency on slow or NFS-backed cache roots; the manifest table is written by every igenerator invocation but never queried by pyi_generator.py; and pyi_generator.py opens the SQLite DB without a table-existence guard, so a misconfigured ITK_WRAP_CACHE path yields an obscure OperationalError rather than a helpful message.

Wrapping/Generators/SwigInterface/igenerator.py (SQLite timeout and manifest table), Wrapping/Generators/Python/itk/pyi_generator.py (DB open without table guard), Wrapping/Generators/CastXML/itk-castxml-cache.py (Windows eviction gap).

Important Files Changed

Filename	Overview
Wrapping/Generators/CastXML/itk-castxml-cache.py	New 636-line two-level content-addressed CastXML cache; logic is well-structured with correct atomic-rename store, gzip compression, multi-root cascade, and L1/L2 key hierarchy. LRU eviction via os.fork() is silently disabled on Windows with no fallback.
Wrapping/Generators/SwigInterface/igenerator.py	Replaces per-class .pkl files with a WAL-mode SQLite DB; adds stamp file output for correct ninja graph ordering. Default 5-second SQLite busy timeout may be too tight for 96 concurrent writers; manifest table is populated but never queried by pyi_generator.py.
Wrapping/Generators/Python/itk/pyi_generator.py	Migrated from file-path-based .pkl loading to SQLite key lookups; correctly reads index.txt for keys then queries pkl_data. DB opened without CREATE TABLE guard — an unexpected ITK_WRAP_CACHE path would give an obscure OperationalError.
Wrapping/macro_files/itk_end_wrap_module.cmake	Adds per-module stamp file as CMake OUTPUT and passes --module_name/--pkl_stamp to igenerator; correct fix for the ninja graph gap where stamp absence now forces igenerator to re-run.
Wrapping/macro_files/itk_auto_load_submodules.cmake	Correctly builds _castxml_cmd to prepend the Python cache wrapper when ITK_WRAP_CASTXML_CACHE is ON; appends cache script to the existing _castxml_depends list which is already referenced in the add_custom_command DEPENDS clause.
CMake/itkWrapCastXMLCacheSupport.cmake	New CMake module adding ITK_WRAP_CASTXML_CACHE option (ON by default) with validation of script path and Python3_EXECUTABLE; cleanly separated from the wrapping core.
.github/workflows/arm.yml	Adds ITK_WRAP_CACHE env var and cache/restore steps for the castxml cache; correctly gated on matrix.python-version presence.
Testing/ContinuousIntegration/AzurePipelinesLinuxPython.yml	Adds Cache@2 task for CastXML cache with per-SHA key and OS/pipeline fallback restore keys; ITK_WRAP_CASTXML_CACHE:BOOL=ON passed to cmake configure.
Testing/ContinuousIntegration/AzurePipelinesMacOSPython.yml	Mirrors the Linux pipeline changes for macOS; cache key correctly uses 'macOSPython' tag to isolate from Linux entries.
Testing/ContinuousIntegration/AzurePipelinesWindowsPython.yml	Same CastXML cache integration for Windows; ITK_WRAP_CACHE uses semicolon-separated path convention handled correctly by the Python scripts.
pyproject.toml	Adds ccache dependency to pixi feature.cxx and introduces configure-python-ci / build-python-ci / test-python-ci tasks with ITK_WRAP_CASTXML_CACHE enabled; clean separation from the existing configure-python task.

Sequence Diagram

%%{init: {'theme': 'neutral'}}%%
sequenceDiagram
    participant CMake as CMake/Ninja
    participant Cache as itk-castxml-cache.py
    participant CastXML as CastXML binary
    participant IGen as igenerator.py
    participant DB as itk-pkl-v1.db SQLite WAL
    participant PyiGen as pyi_generator.py

    CMake->>Cache: invoke with cxx+inc args
    alt L1 HIT and L2 HIT
        Cache-->>CMake: restore output.xml.gz, no subprocess
    else L1 miss, run castxml -E for L2 key
        Cache->>CastXML: castxml -E preprocessor pass
        CastXML-->>Cache: preprocessed output
        alt L2 HIT
            Cache-->>CMake: restore output.xml.gz, refresh L1 map
        else L2 miss
            Cache->>CastXML: full castxml run
            CastXML-->>Cache: output.xml
            Cache-->>CMake: store to L2, write L1 map
        end
    end
    CMake->>IGen: igenerator.py per module, ~96 parallel
    IGen->>DB: WAL transaction DELETE manifest then INSERT pkl_data and manifest
    IGen-->>CMake: write .index.txt with DB keys, touch Module.stamp
    CMake->>PyiGen: pyi_generator.py after stamp dependency
    PyiGen->>DB: SELECT data FROM pkl_data WHERE key equals key
    DB-->>PyiGen: pickled ITKClass bytes
    PyiGen-->>CMake: write .pyi stubs

Loading

%%{init: {'theme': 'base', 'themeVariables': {"darkMode": true, "background": "#0d1117", "primaryColor": "#21262d", "primaryTextColor": "#e6edf3", "primaryBorderColor": "#8b949e", "lineColor": "#8b949e", "textColor": "#e6edf3", "edgeLabelBackground": "#161b22", "actorBkg": "#21262d", "actorBorder": "#8b949e", "actorTextColor": "#e6edf3", "actorLineColor": "#8b949e", "signalColor": "#8b949e", "signalTextColor": "#e6edf3", "noteBkgColor": "#373320", "noteBorderColor": "#d4a72c", "noteTextColor": "#f0e6c0", "labelBoxBkgColor": "#21262d", "labelBoxBorderColor": "#8b949e", "labelTextColor": "#e6edf3", "loopTextColor": "#e6edf3", "activationBkgColor": "#30363d", "activationBorderColor": "#8b949e"}}}%%
sequenceDiagram
    participant CMake as CMake/Ninja
    participant Cache as itk-castxml-cache.py
    participant CastXML as CastXML binary
    participant IGen as igenerator.py
    participant DB as itk-pkl-v1.db SQLite WAL
    participant PyiGen as pyi_generator.py

    CMake->>Cache: invoke with cxx+inc args
    alt L1 HIT and L2 HIT
        Cache-->>CMake: restore output.xml.gz, no subprocess
    else L1 miss, run castxml -E for L2 key
        Cache->>CastXML: castxml -E preprocessor pass
        CastXML-->>Cache: preprocessed output
        alt L2 HIT
            Cache-->>CMake: restore output.xml.gz, refresh L1 map
        else L2 miss
            Cache->>CastXML: full castxml run
            CastXML-->>Cache: output.xml
            Cache-->>CMake: store to L2, write L1 map
        end
    end
    CMake->>IGen: igenerator.py per module, ~96 parallel
    IGen->>DB: WAL transaction DELETE manifest then INSERT pkl_data and manifest
    IGen-->>CMake: write .index.txt with DB keys, touch Module.stamp
    CMake->>PyiGen: pyi_generator.py after stamp dependency
    PyiGen->>DB: SELECT data FROM pkl_data WHERE key equals key
    DB-->>PyiGen: pickled ITKClass bytes
    PyiGen-->>CMake: write .pyi stubs

Loading

_{Reviews (1): Last reviewed commit: "STYLE: Explicitly set ITK_WRAP_CASTXML_C..." | Re-trigger Greptile}

[hjmjohnson]

Yes — this file is the core deliverable of the PR.

itk-castxml-cache.py is a transparent two-level content-addressed wrapper that replaces the bare castxml invocation for all 816 wrapping jobs. CMake/itkWrapCastXMLCacheSupport.cmake points ITK_WRAP_CASTXML_CACHE_SCRIPT at it; Wrapping/macro_files/itk_auto_load_submodules.cmake:207–215 substitutes it for castxml when ITK_WRAP_CASTXML_CACHE=ON. On a warm cache it skips all 816 CastXML subprocesses, saving ~32 min per AzDO Python CI run (−19%) and ~3h 12m on GHA ubuntu-24.04 combined with ccache (−64%).

Documentation/docs/contributing/wrapping_architecture.md has the full picture (Step 1 and the "CastXML cache" section under Caches).

[hjmjohnson]

@dzenanz I'm running this through some stress tests, and trying to simplify it a little bit (I've tried to squeeze small performance increases, but added more complexity for some of the added features). I'm trying to find where the ROI on complexity is worth it. So far big wins come from the castxml cache ( 30+ minutes saved on many CI runs), and preliminary indications are that using a simple SQL db for data store rather than 1000's of small files is going to be a win as well.

This is not ready for review yet...Sorry.

[hjmjohnson]

some minor documentation generalizations.