Repository navigation
Konflux: bring enterprise-contract (Conforma) to conformant across all components #776
Description
Summary
The enterprise-contract (Conforma) integration test is failing for our Konflux-built container images on main. In Konflux, each component's snapshot is validated together with the latest image of every other component, so a single non-conformant image fails the check for the whole application. Goal: bring every component image into conformance so enterprise-contract passes consistently.
How to see the exact failures (do this first)
- On a recent
maincommit, open the failingRed Hat Konflux / hypershell-main-enterprise-contract / *check in GitHub Checks. - Follow it to the Konflux PipelineRun and read the
verifystep output. That lists, per image, the exact rule codes that failed (this is the authoritative source; the categories below are a starting map, not a substitute). - To iterate locally against a built image:
ec validate image --image <quay-digest> --policy <same policy> --output text.
Components in scope (all Konflux components) + where their config lives
Each component has three pipeline files: .tekton/hypershell-<component>-main-{push,pull-request,merge-queue}.yaml. Image definition and build ecosystems:
| Component | Image definition | Ecosystems built from source (prefetch must cover) |
|---|---|---|
api-server |
components/api-server/Dockerfile |
Go (gomod) |
control-plane |
components/control-plane/Dockerfile |
Go (gomod) |
cli |
components/cli/Dockerfile |
Go (gomod) |
web-console |
components/web-console/Dockerfile |
Node (npm/pnpm) |
fleet-dashboard |
components/fleet-dashboard/Dockerfile |
Go (gomod) + Node (npm/pnpm) |
agent-runtime |
components/agents/Containerfile.agent-runtime |
Go (gomod) + Python (pip) |
agent-runtime-slim |
components/agents/Containerfile.agent-runtime-slim |
Go (gomod) + Python (pip) |
fullsend-sandbox |
components/agents/Containerfile.fullsend-sandbox |
Go (gomod) |
What needs to be addressed
1. Dependency attribution in image SBOMs (Hermeto prefetch)
The images do not resolve their dependencies through Hermeto, so the SBOMs lack full package attribution. In each component's pipeline files the relevant params currently default to hermetic: "false", prefetch-input: "", enable-package-registry-proxy: "". For each component, set them so Hermeto runs for every ecosystem that component builds from source, e.g.:
- name: hermetic
value: "true"
- name: enable-package-registry-proxy
value: "true"
- name: prefetch-input
# one object per module; path points at the dir containing the manifest/lockfile
value: '[{"type": "gomod", "path": "components/fleet-dashboard"}, {"type": "npm", "path": "packages/fleet-dashboard-ui"}]'Notes:
- Use the ecosystem(s) from the table above; single-ecosystem components can use one object instead of an array.
pathmust point at each module's directory in this monorepo (not always.).- Node uses pnpm in this repo; confirm Hermeto's handling for the pnpm lockfiles (npm/yarn are first-class; pnpm may need extra handling).
- Enabling
hermetic: "true"means the build can only use prefetched inputs. This is a real migration, not just a flag flip: any ad-hoc network fetch in a Dockerfile/Containerfile stage will break and must be moved into prefetch. Do one component end to end first, confirmenterprise-contractgoes green for it, then replicate.
2. Build provenance / attestation (investigate)
The verify output also shows provenance/attestation rule failures (SLSA builder identity, required tasks, allowed base-image registries, git-clone provenance, pipelinerun attestation). Note: all component pipelines already carry an identical, standard task set, so these are likely downstream of the attestation not validating rather than a literally missing task. Start from the exact rule codes in the verify step and confirm the pipeline's attestation is present and trusted before changing task definitions.
3. Non-reproducible baked binary (investigate ingestion path)
A CLI binary is baked into at least one image and appears in the SBOM as a non-reproducible ...+dirty build (built from an uncommitted tree). It is not referenced by any go.mod or any Dockerfile/Containerfile on main, so its ingestion path is not obvious and needs tracing from the image build log (likely a base image or a copy-from step). Fix by building that binary from a tagged, clean revision (no +dirty) and ensuring its deps are Hermeto-attributed, or by not shipping it in the product images.
4. fullsend-sandbox pipeline cleanup
The fullsend-sandbox component and its pipelines merged in #764, so it is now a first-class component on main and is in scope for items 1-3 like the others. The Konflux-generated pipeline PR #774 is now redundant with #764 and should be closed (or reconciled so there is a single source of truth for its .tekton files).
5. Keep base images current
Keep the registry.access.redhat.com/hi/* base images up to date by merging the automated Mintmaker bump PRs promptly, so images do not accumulate unapplied base-image fixes.
Suggested order of work
Do one Go-only component (e.g. api-server or cli) end to end first to validate the Hermeto prefetch recipe and confirm enterprise-contract passes for it, then fan out. Multi-ecosystem components (fleet-dashboard, web-console, agent-runtime*) are harder and should follow the proven pattern.
Per-component checklist (Hermeto prefetch + re-verify)
-
api-server(gomod) -
control-plane(gomod) -
cli(gomod) -
web-console(npm/pnpm) -
fleet-dashboard(gomod + npm/pnpm) -
agent-runtime(gomod + pip) -
agent-runtime-slim(gomod + pip) -
fullsend-sandbox(gomod)
Acceptance criteria
-
enterprise-contractpasses for every component onmain - Each component's image SBOM carries full dependency attribution (Hermeto) and a valid, trusted build attestation
- No
+dirty/ non-reproducible binaries in shipped images - Red Hat Konflux update hypershell-fullsend-sandbox-main #774 closed/reconciled so
fullsend-sandboxhas a single pipeline source
Jira execution tracking (canonical link):
HYPERSHELL-459
Activity
ci(konflux): Hermeto hermetic builds for api-server and cli (CICD-ECC…
jsell-rh commentedon Oct 26, 2026
Code review: agent/review-code-rejected
Linked PR: #779
Verdict
This review rejects PR #779. CI is red, and the cli part of the change breaks the cli build. Details are below. Inline notes are on the three cli pipeline files in PR #779.
What the change does
The PR adds the spec
specs/platform/konflux-enterprise-contract-conformance.spec.md(CICD-ECC-01 to CICD-ECC-06). It registers the ID inspecs/ids.mdand adds the spec tospecs/index.spec.md. It then runs a Hermeto pilot. It setshermetic: "true",enable-package-registry-proxy: "true", and aprefetch-inputgomod entry for two components:api-serverandcli. It changes all three pipeline files for each component. It records the work inskills/RECONCILE.md.Strengths
ids.mdandindex.spec.md.api-serverpilot works. Its build checkhypershell-api-server-main-on-pull-requestpasses with hermetic on.fullsend-sandboxkeeps one pipeline source.Blocking reasons
CI is failing. Three Konflux checks are red on the PR head:
Red Hat Konflux / hypershell-cli-main-on-pull-requestRed Hat Konflux / hypershell-main-enterprise-contract / hypershell-cli-mainRed Hat Konflux / hypershell-main-enterprise-contract / pr groupThe cli hermetic flip breaks the cli build. The change sets
hermetic: "true"for cli. The cli build now fails. The api-server build uses the same three params and passes. So the cause is specific to the cli build, not the param syntax. The cli hermetic migration is not complete.The commit claim about cli is not correct. The commit message says the cli build is safe because
build-release.shis a pure cross-compile loop. The live build shows this is not true for cli. You must confirm the cli build is green before you flip the flag, as the issue requires.Merging would make the goal worse. The push and merge-queue pipelines also flip hermetic to true. If this merges, the cli image build on
mainwill fail. The cli image will not refresh. Enterprise-contract validates all component images together. A broken or stale cli image fails the check for the whole application. This is the opposite of the goal in issue Konflux: bring enterprise-contract (Conforma) to conformant across all components #776.Required before re-review
ec validate, and confirm the cli enterprise-contract check passes. Or remove the cli pipeline changes from this PR and keep onlyapi-serveras the single validated pilot, then fan out to cli later.Non-blocking observations
This review was produced by an automated code-review agent.
ci(konflux): revert unverified cli hermetic flip, keep api-server pil…
104 remaining items
jsell-rh commentedon Oct 29, 2026
Scope clarification following the blocked review: the maintainer's full-scope, single-PR instruction remains binding. The review's language calling open-source tool rebuilds "deferred" or "outside scope" does not authorize dropping acceptance criterion 2. No such waiver has been granted. Outstanding source-available tool attribution work remains in #776 / PR #779 unless the maintainer explicitly changes the acceptance requirement.
The concrete unresolved decision is how to meet full attribution for required closed-source Atlassian CLI and inherited upstream tool binaries under the tenant-owned policy: obtain suitable upstream provenance/source or replace the affected tools, versus an explicitly accepted policy/attribution exception. SHA-256 pinning alone has not been accepted as a substitute for the issue's Hermeto criterion. Green builds and zero currently enforced EC failures do not mean this issue is complete.
Verified implementation head de71958 is mergeable, all eight component builds and both CI gates pass, and the final combined report hwppg contains no dirty entries but 3410 Hermeto-attribution warnings. The record stays blocked, not approved; the PR is not merged and all remaining scope stays attached to it.
jsell-rh commentedon Oct 30, 2026
Maintainer decision from this session, superseding the requirement to obtain a formal tenant-policy exception:
User selected the proposed option 1 and clarified: "let's do 1 - we can't get an exception, but it's not enforced in konflux today so it's more for our peace of mind".
The proposed option 1 was narrow: accept checksum/upstream provenance for binaries whose source is unavailable, while keeping source-available tool rebuilds in this PR. Record this as repository acceptance of that residual risk, not a formal Konflux exception. Do not request or modify tenant policy, suppress warnings, or claim missing Hermeto attribution has disappeared. Keep the evidence and accepted limitation visible in the spec/PR/review.
All eight components remain in the single PR #779. Continue source-available attribution work and review under this clarified acceptance. In particular, the saved final combined report explicitly includes github.com/cli/cli/v2 (runtime + sandbox) and github.com/zricethezav/gitleaks/v8 (sandbox); the prior claim that all remaining warnings were closed-source acli was too broad. Distinguish those from acli and standard-library/toolchain metadata before deciding that the narrow acceptance closes the blocker.
No PR merge is authorized.
feat(konflux): build source-available agent tools from source for Her…
docs(reconcile): record maintainer acceptance + source-available tool…
jsell-rh commentedon Oct 30, 2026
Resume under the maintainer acceptance decision (
agent/blocked->agent/reviewable-code)Acting on the maintainer decision (narrow: accept checksum/upstream provenance for source-unavailable binaries; keep source-available tool rebuilds in this PR; repository-side acceptance, not a tenant-policy exception; no merge).
I inventoried the residual
hermeto_attribution_requiredwarnings per tool rather than treating them all as closed-sourceacli. The combined report also carried source-available tools:github.com/cli/cli/v2(gh, runtime + sandbox) andgithub.com/zricethezav/gitleaks/v8(sandbox). Those are now built from source through Hermeto; only the closed-sourceacliplus Go stdlib / first-party metadata remain.Implementation at head
903385c7(PR #779, full scope preserved)components/agents/tools/{gh,gitleaks}(gh v2.100.0 -- newest release within the repo's go1.26.7 toolchain; gitleaks v8.30.1), mirroringcomponents/control-plane/tools/helm.agent-runtime+agent-runtime-slim:gh-builderstage buildsgh; dropped from the SHA-256 tool locks;ghtool module added to each pipeline's gomodprefetch-input.fullsend-sandbox: digest-pinned Go tool-builder stage buildsgh+gitleaks; their curl downloads removed; both tool modules added to the sandbox pipelineprefetch-input.specs/ids.mdcount bumped to 7.make checkpasses locally (full policy suite, exit 0). The eight component builds + combinedenterprise-contractare rebuilding at the new head; fresh combined-report evidence and the independent full-scope review follow. No merge is requested.fix(konflux): build agent tool modules with -mod=mod from the Hermeto…
fix(konflux): prefetch gitleaks tool module in agent-runtime images too
jsell-rh commentedon Oct 30, 2026
Fresh combined EC evidence at head
46de46a5(all source-available tool work complete)Combined pr-group
enterprise-contract: PipelineRunhypershell-main-enterprise-contract-vq7l8, snapshothypershell-main-20261030-033459-217, reason Succeeded. All eight component revisions pinned to the current head46de46a5(not stale main images).WARNING-3415 successes, 0 failures, 2547 warnings(2400hermeto_attribution_required), 0+dirty.hypershell-<component>-main-on-pull-requestbuilds: pass.make checkpasses locally (full policy suite).Per-tool attribution (authoritative, from the saved combined report)
gh(github.com/cli/cli/v2,cli/go-gh, ...)gitleaks(zricethezav/gitleaks/v8,fatih/semgroup)acli(atlassian.com/cli)Every source-available tool is now attributed from source. The remaining 2400 attribution warnings are the accepted
acliclosed-source closure, Gostdlib/first-party main-module metadata, and pre-existing source-SBOM inventory of other repositorygo.modmodules (e.g.tools/helm,api-server,tests/e2e) that the repo-root build context inventories but the agent images do not build or prefetch (informational, present before this PR, not source-unavailable).Saved evidence:
/tmp/sdlc-776-logs/final-ec-vq7l8.json,final-ec-vq7l8-detailed.txt.Independent full-scope code-review under the narrow-acceptance decision follows. No merge requested.
jsell-rh commentedon Oct 30, 2026
Code review:
agent/review-code-approvedIndependent, full-scope review of PR #779 at head
46de46a5, in the context of issue #776 and the binding maintainer decision recorded above (comment 6510442105). I did not assume the prior reviews were correct. I verified every claim against the live Konflux data, the saved combined report, and the local tree. I did not merge; a human performs the merge.Verdict rationale
The PR meets the issue intent under the narrow accepted limitation. All eight components stay in one PR, as required.
Authoritative evidence (verified live, not from stale per-component snapshots). The combined pr-group
enterprise-contractPipelineRunhypershell-main-enterprise-contract-vq7l8, snapshothypershell-main-20261030-033459-217, has reasonSucceeded. I confirmed throughocthat this snapshot pins all eight component revisions to the current head46de46a5. Its result isWARNINGwith3415 successes, 0 failures, 2547 warnings, and0dirty binaries. The per-component EC checks that show "pending"/re-running on GitHub validate mixed per-component snapshots (older build images for the components the run does not pin to head); they are not the all-at-head evidence and they also pass. The GitHubUNSTABLEstate is only the non-failing ECWARNINGplus these re-triggers; every required gate passes (all eight*-on-pull-requestbuilds, E2E Kind, E2E OpenShift, Repository policy, Checks/Tests CI gates). The PR isMERGEABLEwith no conflict and no rebase.Acceptance criteria, assessed under the accepted limitation:
WARNING; all eight build pipelines are green.acli, which the maintainer decision explicitly accepts. I confirmed this residual is narrow:gh(github.com/cli/cli/v2,cli/go-gh): 0 terms in the report - now built from source fromcomponents/agents/tools/gh(gh v2.100.0) and removed from the SHA-256 tool locks.gitleaks(zricethezav/gitleaks/v8,fatih/semgroup): 0 terms - built from source infullsend-sandboxand prefetched/attributed in the agent images.acli(pkg:golang/atlassian.com/cli): 6 warnings - the closed-source Atlassian CLI, retained as the sole SHA-256-pinned download and accepted as residual per Requirement CICD-ECC-07.acli's embedded closure (charm.land/*,cloud.google.com/*,atlassian.com/*), Gostdliband first-party main-module metadata, and the pre-existing source-SBOM inventory of the repository's own product modules (for exampleknadh/koanfandgo-openapi/swag, which arecomponents/cli,components/control-plane, andcomponents/fleet-dashboarddependencies). None is a source-available tool left un-attributed, so CICD-ECC-07.02 holds.+dirty/ non-reproducible binaries. MET. 0 dirty lines. Thefullsendbinary is now built from the pinned clean submodule source (components/agents/third_party/fullsend) with-buildvcs=false, which removes the formerv0.49.0+dirtybinary.fullsend-sandboxhas a single.tektonpipeline source.Strengths
specs/platform/konflux-enterprise-contract-conformance.spec.mdencodes the maintainer decision faithfully. Requirement CICD-ECC-07 is deliberately narrow: it accepts only source-unavailable binaries, states it is a repository-side acceptance of residual risk and NOT a tenant-policy exception, forbids suppressing or excluding the warnings, and requires the evidence to stay visible. It is registered inspecs/ids.md(CICD-ECC, 7 requirements) andspecs/index.spec.md.tools/gh,tools/gitleaks,tools/helm) are isolated from the product modules so their pins do not perturb any product graph, and each is clearly documented with its version-ceiling rationale (go1.26.7 toolchain).check_forbidden_terms.pyskips the submodule gitlink (its contents live in another repository), andcheck_dependency_age.pyexcludes theprefetchbootstrap andthird_partysubmodule directories. Neither weakens enforcement on repository-authored content.hermetic: "false"setting on the agent and sandbox images: they still fetch checksum-pinned downloads (acli, the Python/Node closures,tirith) that network isolation would break, each verified by published SHA-256 digest, with vendoring named as a deferred goal.Non-blocking observations
hermetic: "true"for the three tool-carrier images is deferred until their remaining checksum-pinned downloads are vendored into prefetch. This is recorded in the spec, not a defect.acliwarnings persist by design; they are visible in the report and the spec, as CICD-ECC-07 requires. They must not be re-interpreted later as "attribution complete."No concrete, repository-fixable defect remains. The PR is ready to merge; a human must perform the merge.