overlay: opt-in build-time preprocessing of overlay files#166
Open
mobileoverlord wants to merge 1 commit into
Open
overlay: opt-in build-time preprocessing of overlay files#166mobileoverlord wants to merge 1 commit into
mobileoverlord wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
Adds an opt-in build-time preprocessing path for overlay file contents (using the existing {{ ... }} interpolation engine) by materializing a staged overlay tree under .avocado/overlay-staging/, and updates stamp hashing so preprocessed overlay content changes can invalidate builds.
Changes:
- Introduces host-side overlay preprocessing/materialization and a deterministic post-preprocess overlay content digest.
- Wires preprocessing into rootfs/initramfs overlay application and local extension overlay builds (remote extensions warn + skip).
- Extends stamp input hashing to optionally fold in the post-preprocess overlay content digest (plus tests for invalidation behavior).
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| src/utils/stamps.rs | Folds a post-preprocess overlay content digest into stamp hash data when preprocessing is enabled; adds tests. |
| src/utils/overlay_preprocess.rs | New module implementing preprocessing selection, glob matching, digesting, and host-side staging of overlays. |
| src/utils/mod.rs | Exposes the new overlay_preprocess module. |
| src/utils/interpolation/mod.rs | Adds preprocess_text to reuse the interpolation engine for arbitrary file contents. |
| src/commands/rootfs/install.rs | Materializes a staged overlay (when opted-in) before applying overlay in the container. |
| src/commands/ext/build.rs | Adds preprocessing to extension overlay config and stages local overlays before in-container copy. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
e6c6b7f to
8a154ca
Compare
Add `overlay: { preprocess: [globs] | true }` to run the existing
{{ env. }} / {{ config. }} / {{ avocado. }} interpolation over overlay file
*contents* at build time. This lets projects template a secret (such as a
claim token) into overlay files without an in-tree sed + restore dance that
risks leaving secrets in the working tree.
Overlay files are copied verbatim inside the SDK container, so a processed
copy is materialized on the host into `.avocado/overlay-staging/<label>/`
(gitignored scratch, inside the /opt/src bind mount) and the copy is pointed
there. Binary (non-UTF-8) files pass through verbatim; symlinks and mode bits
are preserved. Default off = byte-identical to today.
- interpolation: add preprocess_text (thin wrapper over interpolate_string)
- new utils/overlay_preprocess: PreprocessSpec, glob matcher, materializer,
and a deterministic post-preprocess content digest
- wire into rootfs/initramfs (rootfs/install.rs) and sysext/confext
(ext/build.rs) copy sites; local overlays only (remote-ext overlays warn+skip)
- stamps: fold the post-preprocess content digest into rootfs/initramfs/ext
input hashes when preprocess is on, so a changed template value or an edited
overlay file invalidates the build (verbatim overlays unchanged)
8a154ca to
6151a33
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds opt-in
{{ … }}preprocessing of overlay file contents at build time,so values (e.g. a secret such as an API/claim token) can be templated into
overlay files without mutating the working tree.
Usage
The committed overlay file holds e.g.
token = "{{ env.SOME_TOKEN }}"; thevalue is substituted only into the built image, never back into the tree.
Details
env./config./avocado.interpolation engine via anew
preprocess_text(no new templating language).copy is materialized on the host into
.avocado/overlay-staging/<label>/(gitignored scratch, inside the
/opt/srcbind mount) and the copy is pointedthere.
preserved.
overlays only; remote-extension overlays warn + skip.
input hashes when preprocessing is enabled, so a changed template value or an
edited overlay file invalidates the build. Verbatim overlays are unchanged.
Testing
preprocess_text, glob matching, binary passthrough, and stampinvalidation on a changed template value.
cargo build,cargo test, andcargo clippyare green.