feat(installer): immutable-tag pin + signed sub-script manifest for the curl|bash bootstrap (RFC-0001 R8)

.github/workflows/installer-tests.yaml

@@ -62,13 +62,23 @@ jobs:
           # common.sh and consumed in other sourced files. Warnings are printed
           # below for visibility but don't fail the gate.
           shellcheck --severity=error --shell=bash \
-            scripts/install.sh scripts/install-k8s.sh scripts/lib/*.sh \
+            scripts/install.sh scripts/install-k8s.sh scripts/gen-manifest.sh scripts/lib/*.sh \
             scripts/tests/distro-prereqs.sh scripts/tests/e2e-cluster.sh scripts/tests/e2e-proxy.sh scripts/tests/e2e-auto-upgrade.sh scripts/tests/check-drift.sh
           echo "── shellcheck warnings (advisory, non-blocking) ──"
           shellcheck --severity=warning --shell=bash \
-            scripts/install.sh scripts/install-k8s.sh scripts/lib/*.sh \
+            scripts/install.sh scripts/install-k8s.sh scripts/gen-manifest.sh scripts/lib/*.sh \
             scripts/tests/distro-prereqs.sh scripts/tests/e2e-cluster.sh scripts/tests/e2e-proxy.sh scripts/tests/e2e-auto-upgrade.sh scripts/tests/check-drift.sh || true
 
+      - name: Installer manifest is current (supply-chain, R8)
+        # The bootstrap verifies each sub-script against scripts/manifest.sha256
+        # before running the privileged steps. If a sub-script changed but the
+        # manifest didn't, the release would publish a manifest that rejects the
+        # very scripts it ships — so fail the PR until the manifest is regenerated.
+        # Also cross-checks the bootstrap's FILES list against gen-manifest.sh.
+        run: |
+          chmod +x scripts/gen-manifest.sh
+          scripts/gen-manifest.sh --check
+
       - name: PSScriptAnalyzer (PowerShell installer)
         shell: pwsh
         run: |

.github/workflows/release-helm-chart.yaml

@@ -137,5 +137,130 @@ jobs:
             client-*.tgz
             ingestor-*.tgz
           generate_release_notes: true
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+
+  # ───────────────────────────────────────────────────────────────────────────
+  #  Supply-chain hardening (RFC-0001 R8, backend#889)
+  #
+  #  The curl|bash bootstrap (scripts/install.sh) refuses to run its sub-scripts
+  #  unless it can verify them against a cosign-signed manifest published on THIS
+  #  release. This job produces that manifest, signs it keyless (same Sigstore
+  #  machinery the CLI binary uses), stamps the published installer with this
+  #  immutable tag, and attaches the four assets the bootstrap fetches:
+  #     install.sh            (DEFAULT_REF stamped to this tag)
+  #     manifest.sha256       (sha256 of every sub-script at this tag)
+  #     manifest.sha256.sig   (cosign keyless signature over the manifest)
+  #     manifest.sha256.cert  (the signing certificate to verify .sig against)
+  #
+  #  Verified end-to-end by scripts/tests/install-bootstrap.bats; the release-
+  #  notes verification recipe is in docs/SUPPLY_CHAIN.md.
+  # ───────────────────────────────────────────────────────────────────────────
+  sign-installer-manifest:
+    name: Sign installer manifest (R8)
+    runs-on: ubuntu-latest
+    permissions:
+      contents: write   # attach assets to the release
+      id-token: write   # cosign keyless OIDC
+    steps:
+      - name: Checkout the released tag
+        # Same actions/runner#2788 guard as above — pin to the release tag so the
+        # manifest covers exactly the bytes published at this immutable ref.
+        uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event.release.tag_name }}
+          fetch-depth: 0
+
+      - name: Install cosign
+        uses: sigstore/cosign-installer@v3
+        with:
+          cosign-release: 'v2.4.1'   # keep in lockstep with scripts/install.sh COSIGN_VERSION
+
+      - name: Generate the sub-script manifest
+        # gen-manifest.sh recomputes the manifest from the exact FILES the
+        # bootstrap fetches and cross-checks the two lists are in sync (it
+        # exits non-zero on drift), so a sub-script can never ship unlisted.
+        run: |
+          chmod +x scripts/gen-manifest.sh
+          scripts/gen-manifest.sh
+          echo "── manifest.sha256 ──"
+          cat scripts/manifest.sha256
+
+      - name: Sign the manifest (cosign keyless)
+        env:
+          COSIGN_YES: 'true'
+        run: |
+          cd scripts
+          cosign sign-blob \
+            --output-certificate manifest.sha256.cert \
+            --output-signature   manifest.sha256.sig \
+            manifest.sha256
+          echo "Signed manifest.sha256"
+          ls -l manifest.sha256*
+
+      - name: Stamp the published installer with this immutable tag
+        # The in-repo install.sh carries a placeholder DEFAULT_REF and refuses to
+        # run un-stamped (fail-closed). The published artifact is pinned to THIS
+        # tag so `curl | bash` of the release asset transitively pins every
+        # sub-script. We stamp a copy, never the committed file.
+        #
+        # SECURITY: the tag is passed via env (NOT interpolated into the run:
+        # script). Interpolating `${{ github.event.release.tag_name }}` straight
+        # into a shell assignment lets a tag containing a single quote break out
+        # and inject shell into THIS job — which holds id-token:write +
+        # contents:write, the trust-establishing privileges. As an env var it is
+        # plain data; the shell never re-parses it (RFC-0001 R8, backend#889).
+        env:
+          TAG: ${{ github.event.release.tag_name }}
+        run: |
+          install -m 0755 scripts/install.sh scripts/install.stamped.sh
+          # Replace only the placeholder token; fail loudly if it's absent so a
+          # refactor that drops it can't silently publish an un-pinned installer.
+          grep -q '__TRACEBLOC_RELEASE_REF__' scripts/install.stamped.sh \
+            || { echo "::error::DEFAULT_REF placeholder not found in install.sh"; exit 1; }
+          # Robust substitution: do a literal find/replace with awk instead of
+          # `sed s|...|...|`, so a tag containing sed-special bytes ('/' as the
+          # delimiter, '&' as match-text, '\' as an escape) can't corrupt the
+          # stamp or inject regex metacharacters. The index()/substr() loop treats
+          # BOTH the placeholder and the replacement (read from ENVIRON, never the
+          # command line) as plain strings — no regex, no backreference, no
+          # delimiter — so every byte of the tag lands verbatim.
+          TAG="$TAG" awk '
+            BEGIN { ph = "__TRACEBLOC_RELEASE_REF__"; rep = ENVIRON["TAG"] }
+            {
+              out = ""; line = $0
+              while ((p = index(line, ph)) > 0) {
+                out = out substr(line, 1, p - 1) rep
+                line = substr(line, p + length(ph))
+              }
+              print out line
+            }
+          ' scripts/install.stamped.sh > scripts/install.stamped.sh.new
+          mv scripts/install.stamped.sh.new scripts/install.sh
+          rm -f scripts/install.stamped.sh
+          # Confirm the stamp landed exactly and the placeholder is fully gone.
+          grep -n 'DEFAULT_REF=' scripts/install.sh
+          if grep -q '__TRACEBLOC_RELEASE_REF__' scripts/install.sh; then
+            echo "::error::placeholder still present after stamping"; exit 1
+          fi
+
+      - name: Self-test the stamped installer parses and refuses bad refs
+        run: |
+          bash -n scripts/install.sh
+          # Un-pinned dev ref without the opt-in must fail closed (exit non-zero).
+          if BRANCH=some-branch bash scripts/install.sh </dev/null >/dev/null 2>&1; then
+            echo "::error::stamped installer did not fail closed on a mutable BRANCH"; exit 1
+          fi
+          echo "stamped installer fails closed on a mutable ref (expected)"
+
+      - name: Attach installer + signed manifest to the release
+        uses: softprops/action-gh-release@v2
+        with:
+          tag_name: ${{ github.event.release.tag_name }}
+          files: |
+            scripts/install.sh
+            scripts/manifest.sha256
+            scripts/manifest.sha256.sig
+            scripts/manifest.sha256.cert
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

README.md

@@ -68,7 +68,7 @@ bash <(curl -fsSL https://tracebloc.io/i.sh)
 irm https://tracebloc.io/i.ps1 | iex
 ```
 
-The installer pulls helper scripts from this repo at runtime — see [`scripts/install-k8s.sh`](scripts/install-k8s.sh) and [`scripts/install-k8s.ps1`](scripts/install-k8s.ps1).
+The installer pulls helper scripts from this repo at runtime — see [`scripts/install-k8s.sh`](scripts/install-k8s.sh) and [`scripts/install-k8s.ps1`](scripts/install-k8s.ps1). Those scripts are pinned to an **immutable release tag** and each is **verified against a cosign-signed manifest** before it runs; the install **fails closed** if verification can't complete (it never silently runs unverified code). See [docs/SUPPLY_CHAIN.md](docs/SUPPLY_CHAIN.md) for the integrity model and how to verify a release by hand.
 
 ### Helm install
 

docs/SECURITY.md

@@ -51,6 +51,14 @@ The following parts of the system are treated as trusted and are **not** in scop
 - Tracebloc engineers publishing the training base images (`tracebloc/*-cpu`, `tracebloc/*-gpu`) and the chart artifact.
 - The Helm chart itself and the values the customer provides at install time.
 
+> **Delivery integrity is verified, not assumed.** Trusting "the chart artifact"
+> and "the installer scripts" means trusting the bytes that actually reach the
+> box — so the `curl | bash` bootstrap pins to an immutable release tag and
+> verifies every sub-script against a cosign-signed manifest before running the
+> privileged steps (credential mint/write, Helm). See [§4.8](#48-installer-supply-chain-integrity-g8)
+> and [docs/SUPPLY_CHAIN.md](SUPPLY_CHAIN.md). This addresses RFC-0001 R8 (the
+> dominant supply-chain gap for a regulated buyer), tracked as backend#889.
+
 ### 1.3 Untrusted components
 
 - **The Python file, weight file, and training plan submitted by an external data scientist.**
@@ -225,6 +233,37 @@ pod-security.kubernetes.io/audit: restricted
 
 `enforce: restricted` is on by default on CSI-backed deployments (EKS/AKS/OC); bare-metal overrides it off via `ci/bm-values.yaml`. See §6.6 and §8.5.
 
+### 4.8 Installer supply-chain integrity (G8)
+
+**Threat.** The `curl | bash` bootstrap (`scripts/install.sh`) is the most
+privileged code in the product: the sub-scripts it fetches mint the machine
+credential, write it to disk, and run Helm as a cluster admin. If the bytes that
+reach the box can be altered — by moving a mutable branch ref or by an on-path
+attacker — none of the in-cluster defenses above matter, because the attacker
+runs first, as root.
+
+**Mechanism (RFC-0001 R8, backend#889):**
+
+- **Immutable ref.** The bootstrap fetches every sub-script from a pinned release
+  **tag** (content-addressable), never a mutable branch. A branch / non-`vX.Y.Z`
+  ref is refused unless an operator sets `TRACEBLOC_ALLOW_UNVERIFIED=1` (a loud,
+  developer-only escape hatch).
+- **Signed manifest.** Each sub-script's sha256 is checked against a
+  `manifest.sha256` published on the release. A tampered sub-script, or one
+  missing from the manifest, **aborts the install before any privileged step**.
+- **Authenticated + fail-closed.** The manifest is verified with a **cosign
+  keyless signature** (same Sigstore chain as the CLI binary). If cosign is
+  absent the bootstrap bootstraps a pinned, checksum-verified cosign; if it can
+  neither find nor bootstrap cosign it **fails closed** rather than degrading to
+  a same-channel checksum.
+
+**Where implemented:** `scripts/install.sh` (verification), `scripts/gen-manifest.sh`
+(manifest generation), `.github/workflows/release-helm-chart.yaml`
+(`sign-installer-manifest` job: generate + sign + stamp + attach), and the CLI's
+own installer for the leaf binary (`tracebloc/cli` `scripts/install.sh`, made
+mandatory under the same ticket). Full model + release-pipeline + key-management
+follow-ups: [docs/SUPPLY_CHAIN.md](SUPPLY_CHAIN.md).
+
 ---
 
 ## 5. Per-platform caveats
@@ -457,6 +496,19 @@ If the customer enables the policy on a CNI that doesn't enforce (default EKS, F
 
 A malicious model can allocate memory / consume CPU up to the pod's resource limits. `resources.limits` are applied (defaults `cpu=2,memory=8Gi`). A pod running at 100% of its limits is expected behavior for training; OOMKill or eviction is the Kubernetes-native response. The chart does not attempt to detect or prevent resource-intensive pathological inputs.
 
+### 8.9 Cosign-bootstrap trust root on boxes without cosign — **security / operator**
+
+When cosign is not already installed, the bootstrap (§4.8) downloads a pinned
+cosign from the sigstore GitHub release and verifies it against
+`cosign_checksums.txt` fetched over the **same TLS channel**. This is a pragmatic
+trust root — strictly better than the previous "no verification at all" — but it
+is not signature-rooted (you can't verify cosign's signature without cosign). A
+sufficiently capable on-path attacker who can also forge a TLS chain to GitHub
+could substitute both. **Mitigation for regulated buyers:** pre-install cosign
+from the OS package manager (or an internal mirror) before running the
+installer, so the bootstrap uses a cosign you already trust. Documented in
+[docs/SUPPLY_CHAIN.md §6](SUPPLY_CHAIN.md#6-operator-guidance--verifying-a-release-by-hand).
+
 ---
 
 ## 9. If you suspect compromise