[GLUTEN-12288][VL] Publish a prebuilt x86 static Velox image by FelixYBW · Pull Request #13447 · apache/gluten

FelixYBW · 2026-10-25T08:46:05Z

What changes are proposed in this pull request?

Part of #12288. Implements the publishing/key stage of the prebuilt Velox proposal, starting with the amd64 AlmaLinux 8/GCC 13 static variant. Native CI consumers remain unchanged; switching them to the image (the slot holding their key, else velox-static-latest, which only needs an incremental Velox rebuild) is a follow-up after the image is measured and published.

Add a shared key/build utility and Dockerfile that compile Velox into /opt/velox. Build flags come directly from the existing x86 static CI script. Preserve the full Velox build tree and vcpkg dependencies at /opt/gluten/dev/vcpkg, since generated CMake files contain absolute paths.
The key covers the resolved Velox commit/repository, tracked Velox scripts and patches, vcpkg ports/configuration, build scripts, recipes, dev/docker/install-ci-tools.sh (now run by the base image), base image digest, and build variant/environment. Generated build outputs do not affect it. Fetch the resolved commit explicitly and verify the key before building.
Publish to a fixed set of 8 tags, apache/gluten:velox-static-slot-0 to -7, plus velox-static-latest and velox-static-almalinux-8-gcc13 aliases. CI can't delete Docker Hub tags, so a tag per key would grow without bound (~0.6 key changes/day plus weekly base refreshes); a new key instead overwrites the least recently published slot. Each image records its key and publish order in the org.apache.gluten.velox-key and org.apache.gluten.velox-sequence labels, read from the manifest without pulling. An existing key is reused without rebuilding. The key and input metadata are also stored in /opt/velox/.velox-key and .velox-build.json.
Add a native-input publishing workflow so merged Velox/vcpkg changes do not rebuild every unrelated Docker image. docker_image.yml also calls it after the refreshed AlmaLinux base is published, including weekly refreshes. Only apache/gluten on main can publish, using the existing Docker Hub secrets; PRs run inexpensive key tests only.
Before any image push, load it locally, report its uncompressed size and runner disk usage, and time a Gluten-only build_gluten_cpp smoke build with NUM_THREADS=2. Publish only after that build succeeds.
Document the image, key identity, slot rotation, measurement step, and local build/smoke commands.

How was this patch tested?

All 21 Python unit/integration tests passed: cache invalidation for each input, slot selection (reuse, empty slot first, least recently published, unlabelled slots), label parsing for single- and multi-platform manifests, missing-slot vs. inspect-error handling, base/commit changes, generated-file exclusion, copied checkout identity, actual Git annotated-tag resolution, CLI JSON/GitHub-output agreement, invalid inputs, and pre-build key mismatch handling.
Actionlint passed for both publisher workflows, including the reusable workflow's declared secrets, before the slot rotation change; the updated velox_image.yml has only been checked for valid YAML. The slot step's docker buildx imagetools inspect call is unit-tested with mocked output and first runs for real on the first publish.
Black, license headers (including the normally excluded new dev files), and git diff --check passed.
Exercised key generation against the real configured Velox remote and the published Docker Hub base digest; it resolved dft-2026_10_23 to 0a477fa804b9424c8ce251bd8b1f60b1bdafd11c and hashed 129 tracked inputs.

Draft: Docker image build and performance measurements are pending. This Windows workspace has neither Docker nor WSL, so no image-size, runner-disk, or Gluten-only timing claim is made. The publisher enforces the smoke build before publication, and the documented commands allow those measurements on an amd64 Linux Docker host. Keep this draft until those results are available; existing native CI has not been switched to consume the image.