[Kevin Cho (chokevin)]

Signal now treats MPIJob as a first-class production certification backend. For each MPIJob check, one launcher coordinates one worker per immutable participant, captures typed launcher/worker evidence, and cannot complete until retained workers and the full MPIJob ownership tree are gone. One immutable CertificationRun executes exactly 15 ordered checks through one serial Argo Workflow; Pod remains available as the simpler per-participant backend.

Dependency: This PR is blocked on #1001, which publishes the baseline project-signal-runner image. The runner implementation is intentionally not duplicated here. Both PRs use fork branches, so GitHub cannot represent them as a native stack; #963 remains targeted at main and must be rebased after #1001 merges.

• Exactly two Signal CRDs: cluster-scoped operator-authored CertificationProfile and namespaced controller-generated immutable CertificationRun.
• Standalone execution through one fixed adapter WorkflowTemplate. MPIJob creates one coordinated job with one launcher and one worker per immutable participant; Pod creates one independently pinned Pod per participant.
• One explicit participant policy selects the complete immutable Node group. Empty all-Node selectors, roles, topology assignment, subgroups, and per-check participant filtering are intentionally absent.
• Participant eligibility is derived from the largest effective per-Node resource request across the 15 checks, including canonical GPU counts.
• The Profile pins one operator-owned immutable source ConfigMap by name and aggregate SHA-256 digest. Admission verifies it through the authoritative API reader before participant selection or Node mutation, then copies the exact bytes into a Run-owned immutable snapshot.
• Each check specifies only scriptKey, backend, gpuCount, backendActiveDeadline, and a constrained workload policy. The suite deadline is derived from those 15 immutable deadlines plus an evidence reserve.
• Signal owns reservation and cordon state. Success releases automatically; Failed, Inconclusive, and Canceled Runs remain cordoned until monotonic spec.release: true.
• Controller and adapter use separate least-privilege identities. Their control Pods receive the API tokens required by their in-cluster clients; adapter-created backend Pods explicitly deny token automount.
• Run status remains compact. Measurements, backend identities, logs, states, and timestamps live only in the digest-bound durable evidence document.
• Detailed evidence is persisted before teardown. Uncertain or retained backends are quiesced before later checks can start.
• Production-only controller surface: no tracked Signal test files were added.

Evidence

MPIJob cleanup and live canary: 20-second fframes animation showing the retained-worker defect, the exact ownership/UID-safe cleanup change, launcher/worker SSH execution, and the final evidence/cleanup/release state.

signal-mpi-first-class-20s.mp4

A bounded live cutover was completed in a private test environment with digest-pinned controller, adapter, and runner images.

The Pod canary was used first as a control-plane baseline for admission, Argo startup, identity, evidence persistence, cleanup, and release; it was not the main backend acceptance proof. It exposed a deployment defect: the adapter ServiceAccount denied token automount, so the Argo executor and adapter could not obtain in-cluster Kubernetes configuration. The Run correctly became Inconclusive, persisted evidence, cleaned its workload, and retained its reservation until explicit release. Commit b9fe9eba1 enables the adapter Workflow identity's required API token; backend workload Pods remain tokenless. The corrected Pod canary then passed all 15 ordered checks with complete evidence, cleanup, and automatic release.

The first useful MPI execution exposed a separate cleanup defect. cleanPodPolicy: None intentionally retained the running SSH worker for evidence, but foreground MPIJob deletion inherited the worker's ordinary grace period and outlived the adapter's 30-second quiescence fence. Commit 0d5413beb fixes the root cause: after evidence is durable, the adapter discovers only worker Pods matching the Signal Run label, MPIJob UID ownership, and worker container identity; deletes those exact Pods with UID preconditions and zero grace; then foreground-deletes the MPIJob and waits for its complete ownership tree to disappear.

The corrected mixed canary completed all 15/15 ordered checks:

• R01 created one MPIJob with one launcher and one worker on the immutable single-Node participant.
• The worker accepted public-key SSH from the launcher with zero restarts.
• The launcher completed successfully, emitted a valid typed Signal result, and verified eight ranks, world size eight, one host, and eight visible GPUs.
• R02-R15 completed the previously proven Pod-backed GPU inventory path.
• Every check ended Succeeded / Pass; the Run ended Succeeded with AllChecksPassed.
• Evidence digest sha256:893d2dc235d6cdead37bc7ceb4911d966fb4ee9e4711bcdca321ced5bc99c351 independently matched the uncompressed evidence document.
• WorkloadCleanupComplete=True and Released=True; no Run-owned Workflow, MPIJob, Job, Pod, or script snapshot remained, and the participant was schedulable again.
• The temporary canary Profiles and source script ConfigMaps were deleted; terminal Runs and immutable evidence remain for review.

Verification

Current head e9eb9d174ad193b5102e9793de4bfa43584d0bbb:

• go test ./internal/signaladapter ./cmd/project-signal-adapter
• go vet ./internal/signaladapter ./cmd/project-signal-adapter
• git diff --check
• Live Pod canary: 15/15 passed with digest-bound evidence, cleanup, and release.
• Live mixed MPIJob/Pod canary: 15/15 passed with launcher/worker SSH, eight-rank execution, independently matched evidence, cleanup, and release.
• Exact-head GitHub validation is green: 38 checks passed and one non-applicable update job skipped. CI, CodeQL, gantry e2e, playpen e2e, and racer e2e workflows all completed successfully.

Remaining limits and review guide

The MPI canary proves single-Node launcher/worker orchestration and process launch. It does not prove cross-Node networking, multi-Node placement, distributed collectives, CUDA toolkit workloads, NCCL communication, legacy-contract compatibility, regression testing, or soak behavior.

The external participant lacked persistent private-registry authentication, so the exact runner digest was preloaded for this bounded canary. Another uncached external participant still needs a persistent registry credential solution or an explicit exact-digest preload. #1001 remains the source and publication dependency for the baseline runner image.

Review api/signal/v1alpha1/ for the two CR contracts and canonical bundle digest; internal/signal/controller.go, evidence.go, and projection.go for admission, recovery, audit, cleanup, and release; internal/signaladapter/adapter.go and runner.go for Pod/MPIJob execution and evidence-first quiescence; and deploy/signal/ for least-privilege RBAC, token boundaries, templates, image contracts, and lifecycle documentation. Review the baseline execution image separately in #1001.

AI assistance: GitHub Copilot assisted with implementation, investigation, validation, video preparation, and PR drafting. Independent review used Claude Opus 5.5 and GPT-6 Astra. The author verified the claims and final text; no human approval is implied.

[Kevin Cho (chokevin)]

Typed workload policy, Pod queue labeling, MPI worker isolation, and focused contract tests (15 seconds):

signal-workload-policy-evidence.mp4

[Kevin Cho (chokevin)]

Copilot-generated 20-second fframes explainer. The source excerpts remain present in the production cleanup fixes. The recorded integration is explicitly archival, before the final production-only trim: 15 task nodes, 10 Pass and 5 Inconclusive. This is not a new live run or evidence that the updated head passed certification. The updated implementation is being validated separately; the live cluster and soak were not modified.

signal-pr963-20s.mp4

[Kevin Cho (chokevin)]

Updated 20-second fframes architecture explainer with local narration: why Signal is needed, how the controller/CRDs/RBAC/adapter template are deployed alongside Argo, operator-owned scripts and Profile, the exact Node request label, immutable admission, serial tasks, and script execution in Pods/MPIJobs. This is an architecture animation, not a live execution recording. Placeholder deployment commands were not executed. It supersedes the archival clip as the architecture overview, not as runtime validation.

signal-deployment-trigger-20s.mp4