Skip to content

overlay: opt-in build-time preprocessing of overlay files#166

Open
mobileoverlord wants to merge 1 commit into
mainfrom
jschneck/overlay-preprocess
Open

overlay: opt-in build-time preprocessing of overlay files#166
mobileoverlord wants to merge 1 commit into
mainfrom
jschneck/overlay-preprocess

Conversation

@mobileoverlord

Copy link
Copy Markdown
Contributor

Summary

Adds opt-in {{ … }} preprocessing of overlay file contents at build time,
so values (e.g. a secret such as an API/claim token) can be templated into
overlay files without mutating the working tree.

Usage

overlay:
  dir: overlay
  preprocess:                 # opt-in; default off = byte-identical to today
    - "etc/some/config.toml"  # globs, or `preprocess: true` for all UTF-8 files

The committed overlay file holds e.g. token = "{{ env.SOME_TOKEN }}"; the
value is substituted only into the built image, never back into the tree.

Details

  • Reuses the existing env. / config. / avocado. interpolation engine via a
    new preprocess_text (no new templating language).
  • Overlay files are copied verbatim inside the SDK container, so a processed
    copy is materialized on the host into .avocado/overlay-staging/<label>/
    (gitignored scratch, inside the /opt/src bind mount) and the copy is pointed
    there.
  • Binary (non-UTF-8) files pass through verbatim; symlinks and mode bits are
    preserved.
  • Wired into the rootfs/initramfs and sysext/confext overlay copy sites. Local
    overlays only; remote-extension overlays warn + skip.
  • Stamps fold a post-preprocess content digest into the rootfs/initramfs/ext
    input hashes when preprocessing is enabled, so a changed template value or an
    edited overlay file invalidates the build. Verbatim overlays are unchanged.

Testing

  • Unit tests for preprocess_text, glob matching, binary passthrough, and stamp
    invalidation on a changed template value.
  • cargo build, cargo test, and cargo clippy are green.

Copilot AI review requested due to automatic review settings July 3, 2026 14:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds an opt-in build-time preprocessing path for overlay file contents (using the existing {{ ... }} interpolation engine) by materializing a staged overlay tree under .avocado/overlay-staging/, and updates stamp hashing so preprocessed overlay content changes can invalidate builds.

Changes:

  • Introduces host-side overlay preprocessing/materialization and a deterministic post-preprocess overlay content digest.
  • Wires preprocessing into rootfs/initramfs overlay application and local extension overlay builds (remote extensions warn + skip).
  • Extends stamp input hashing to optionally fold in the post-preprocess overlay content digest (plus tests for invalidation behavior).

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
src/utils/stamps.rs Folds a post-preprocess overlay content digest into stamp hash data when preprocessing is enabled; adds tests.
src/utils/overlay_preprocess.rs New module implementing preprocessing selection, glob matching, digesting, and host-side staging of overlays.
src/utils/mod.rs Exposes the new overlay_preprocess module.
src/utils/interpolation/mod.rs Adds preprocess_text to reuse the interpolation engine for arbitrary file contents.
src/commands/rootfs/install.rs Materializes a staged overlay (when opted-in) before applying overlay in the container.
src/commands/ext/build.rs Adds preprocessing to extension overlay config and stages local overlays before in-container copy.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/utils/overlay_preprocess.rs
Comment thread src/utils/overlay_preprocess.rs Outdated
Comment thread src/utils/overlay_preprocess.rs
Comment thread src/utils/overlay_preprocess.rs
Comment thread src/utils/stamps.rs
@mobileoverlord mobileoverlord force-pushed the jschneck/overlay-preprocess branch from e6c6b7f to 8a154ca Compare July 3, 2026 15:23
Add `overlay: { preprocess: [globs] | true }` to run the existing
{{ env. }} / {{ config. }} / {{ avocado. }} interpolation over overlay file
*contents* at build time. This lets projects template a secret (such as a
claim token) into overlay files without an in-tree sed + restore dance that
risks leaving secrets in the working tree.

Overlay files are copied verbatim inside the SDK container, so a processed
copy is materialized on the host into `.avocado/overlay-staging/<label>/`
(gitignored scratch, inside the /opt/src bind mount) and the copy is pointed
there. Binary (non-UTF-8) files pass through verbatim; symlinks and mode bits
are preserved. Default off = byte-identical to today.

- interpolation: add preprocess_text (thin wrapper over interpolate_string)
- new utils/overlay_preprocess: PreprocessSpec, glob matcher, materializer,
  and a deterministic post-preprocess content digest
- wire into rootfs/initramfs (rootfs/install.rs) and sysext/confext
  (ext/build.rs) copy sites; local overlays only (remote-ext overlays warn+skip)
- stamps: fold the post-preprocess content digest into rootfs/initramfs/ext
  input hashes when preprocess is on, so a changed template value or an edited
  overlay file invalidates the build (verbatim overlays unchanged)
@mobileoverlord mobileoverlord force-pushed the jschneck/overlay-preprocess branch from 8a154ca to 6151a33 Compare July 4, 2026 13:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants