Generalize the WebApp abstraction (ascoachingogvaner, wedding-app)

[devantler]

Summary

Generalize a WebApp archetype — a reusable definition of a containerized web application (workload + route + TLS + scaling + secrets) — so the tenant web apps (ascoachingogvaner, wedding-app) and future ones become a small per-app declaration instead of copied workload boilerplate.

Sibling to the Tenant archetype issue: a Tenant sets up the namespace/isolation/sync; a WebApp defines the application that runs inside it.

Context

ascoachingogvaner (coaching site) and wedding-app are simple web apps. Their workload manifests (Deployment/container or chart, Service, Gateway-API HTTPRoute, optional scaling, optional secrets) live in each tenant's own OCI artifact (synced via the platform-provisioned OCIRepository + Flux Kustomization), not in this repo — but they follow the same web-app shape and will multiply as more tenants onboard. The shape is near-identical: image/chart + host + TLS + a route to the central gateway + (optionally) auth and a secret.

Goal & constraint

One WebApp abstraction → a per-app instance specifying just {image|chart, host, tls, scaling, auth?, secretRefs?}. No more Kustomize overlay/component stacking (the readability reason we're choosing an archetype).

Options to evaluate

1. Helm app/library chart — recommended start. A webapp chart (named templates for Namespace?/Deployment/Service/HTTPRoute/optional HTTPScaledObject/optional ExternalSecret) consumed via a HelmRelease or referenced from the tenant artifact. values.schema.json gives light typing; stays inside the Flux→HelmRelease model; zero new controllers; no pre-1.0 risk.
2. KRO WebApp RGD. Typed, kubectl-native, conditional children via includeWhen (KEDA/OIDC). But KRO is pre-1.0 (v1alpha1) and adds an alpha controller at the infrastructure-controllers chokepoint — see the KRO assessment. Better revisited once KRO hits beta/1.0.

Open design question — placement

Tenant workloads live in tenant-owned OCI artifacts. Decide whether the WebApp archetype is:

• a platform-published chart/RGD that tenants reference from their artifact (platform owns the template, tenant owns the values), or
• defined per-tenant.

The platform-published option keeps the archetype DRY across tenants and lets us evolve it centrally; it should be the default unless there's a reason to fork per tenant.

Platform constraints any solution must respect

• HTTPRoute attaches to the central Cilium Gateway in kube-system (reference it, don't recreate it).
• Optional KEDA HTTP scale-to-zero mirrors the existing 5-point recipe (interceptor repoint + netpol + initial replicas 0).
• Kyverno mutate policies apply to the rendered pods (security context, anti-affinity, spread, etc.) — the archetype shouldn't fight them.
• Secrets via ESO + SOPS, never inline.
• enforce-flux-best-practices (Enforce) requires interval/timeout/retries on any templated HelmRelease.
• ksail workload validate builds every kustomization standalone (skip-kinds for any new CRD-backed instance).

Acceptance criteria

• Decision recorded (ADR, or reuse the Tenant-archetype decision) on Helm chart vs KRO, including archetype placement (platform-published vs per-tenant).
• One app (ascoachingogvaner or wedding-app) migrated as a pilot; behavior-preserving where applicable.
• A new web app becomes a small declaration.
• No added Kustomize overlay/component stacking.

Related

• Composes with the Tenant archetype — see the sibling issue.
• Incremental DRY already shipped: refactor(infrastructure): hetzner tracks controller aggregate, not a hand list #1927, refactor(infrastructure): centralize HelmRelease Flux remediation defaults #1928, refactor(infrastructure): use path-only kustomize patches where self-identifying #1929, style(kustomize): add yaml document-start markers + normalize list indent #1931.

---

🤖 Filed from an architecture discussion with Claude Code. Source: kro.run.

[devantler]

Sibling archetype (composes with this one): #1932

[devantler]

🤖 Generated by the Daily AI Assistant

ADR (Proposed) — WebApp archetype: Helm chart vs KRO, and where it lives

Delivering acceptance-criterion 1 ("Decision recorded on Helm chart vs KRO, including archetype placement (platform-published vs per-tenant)") as a Proposed record for your ratification. Status flips to Accepted when you promote it (or redirect). This composes with — and deliberately reuses the mechanism of — the Tenant archetype decision (#1932, Helm chart recommended).

The WebApp question has a twist the Tenant question doesn't: a Tenant registration skeleton lives in this repo (k8s/bases/apps/<tenant>/), but a WebApp workload lives in the tenant's own OCI artifact (oci://ghcr.io/devantler-tech/<tenant>/manifests, cosign-verified, synced by the platform-provisioned OCIRepository+Kustomization — verified in k8s/bases/apps/{ascoachingogvaner,wedding-app}/sync.yaml). So "where the archetype lives" is the load-bearing decision here, not an afterthought.

Decision drivers (weighted by this platform's reality)

1. No new controller/CRD in infrastructure-controllers — the recurring wedge point (same driver 1 as Generalize the Tenant abstraction — evaluate Capsule vs KRO vs Helm library chart #1932). helm-controller is already cluster-wide; adding an operator here is the highest-cost move.
2. DRY across tenants with central evolution — the shape ({image|chart, host, tls, scaling?, auth?, secretRefs?}) is near-identical across ascoachingogvaner, wedding-app, and every future webapp; the template must live in one place we can version, not be re-copied into each tenant repo.
3. Honor the workload-decoupling boundary — tenants own their artifact + values; the platform owns trust (cosign), isolation (Flux impersonation, Kyverno, netpol) and — proposed here — the template. Don't drag tenant workload definitions back into this repo.
4. Respect the verified platform constraints (all confirmed against the repo): one central Cilium Gateway named platform in kube-system (HTTPRoutes attach via parentRefs:[{name: platform, namespace: kube-system, sectionName: https}], e.g. k8s/bases/apps/actual-budget/httproute.yaml); the KEDA HTTP scale-to-zero recipe; Kyverno mutate policies on rendered pods; ESO+SOPS secrets (never inline); enforce-flux-best-practices (Kyverno Enforce); ksail workload validate standalone-build parity.
5. Match scale — ~2 webapps today, growing slowly; behavior-preserving, render-diffable migration.

Evaluation

KRO WebApp RGD (build, typed) — REJECTED (same as #1932). Pre-1.0 v1alpha1; lands an alpha controller + dynamic CRDs in the wedge-prone infrastructure-controllers layer (driver 1 ✗). The typed/kubectl-native + includeWhen conditional-children upside is real but doesn't pay for the layer risk at 2-app scale. Revisit post-1.0 with a concrete typed-API need — the conditional KEDA/auth children are exactly KRO's sweet spot, so it's the right thing to re-evaluate later.

Helm chart — RECOMMENDED, as a platform-published OCI chart. Resolving the open placement question explicitly:

• Placement = platform-published, NOT per-tenant. Publish a webapp application chart as a cosign-signed OCI chart to ghcr.io/devantler-tech/charts/webapp (mirroring the existing per-tenant publish-app OIDC+cosign trust model). Each tenant's artifact then carries a thin HelmRelease (or OCIRepository→HelmRelease pair) pointing chart.spec.sourceRef at that chart with a per-app values block. Platform owns the template; tenant owns the values — the issue's own default, and the only option that gives DRY + central evolution (driver 2 ✓) without forking the template per tenant (driver 3 ✓). Per-tenant charts are rejected: they re-fork the very boilerplate we're removing.

• No new controller (driver 1 ✓): helm-controller already reconciles HelmReleases platform-wide. The tenant's Kustomization simply emits a HelmRelease instead of raw manifests; nothing new enters infrastructure-controllers.

• One indirection, not overlay stacking (acceptance ✓): chart → values, flatter than overlay-on-overlay, and Helm is already in the platform vocabulary.

• Templates the whole webapp shape (driver 2 ✓): Deployment, Service, HTTPRoute (with the central-Gateway parentRef baked in), and — behind value conditionals — the KEDA scale-to-zero set and an ExternalSecret. A new webapp becomes a ~6-line values block.

• Constraints honored verbatim (driver 4 ✓):

  • Gateway: HTTPRoute.parentRefs hard-wired to platform/kube-system/https; the app only supplies host.
  • KEDA scale-to-zero (values.scaleToZero, default off): templates the verified 5-point recipe — (1) initial replicas, (2) HTTPScaledObject (min/max, scaledownPeriod, request-rate metric), (3) HTTPRoute backend repointed to keda-add-ons-http-interceptor-proxy.keda.svc:8080, (4) CiliumNetworkPolicy ingress from the keda namespace, (5) drain-safe PodDisruptionBudget. Honest caveat: the interceptor lives in the keda namespace and the cross-namespace ReferenceGrant (k8s/bases/infrastructure/controllers/keda-http-add-on/reference-grant.yaml) enumerates specific app namespaces — so enabling scale-to-zero for a new tenant namespace needs a one-line platform-side ReferenceGrant addition (it can't be templated from the tenant artifact). The chart should emit a clear comment / values note pointing at that step.
  • Kyverno mutate: the chart must not fight the mutate policies — leave securityContext / anti-affinity / topology-spread to Kyverno rather than hardcoding conflicting values, mirroring how existing apps rely on the postRenderers + mutate combo.
  • Secrets: values.secretRefs reference an ESO ExternalSecret/SecretStore (and SOPS for the wedding-app case) — never inlined.
  • enforce-flux-best-practices (Enforce): the generated HelmRelease must template spec.interval, spec.timeout, spec.install.remediation.retries, spec.upgrade.remediation.retries (≥1/-1), and spec.upgrade.remediation.remediateLastFailure: true (bounded, known — values defaults cover it).

• Honest caveats specific to WebApp (beyond Generalize the Tenant abstraction — evaluate Capsule vs KRO vs Helm library chart #1932's):

  1. This is the platform's first first-party Helm chart. Today every app sources an external chart (actual-budget→community-charts, whoami→cowboysysop, etc. — no Chart.yaml in-repo). So this adds a new chart-build-and-publish pipeline (a publish-chart workflow + cosign signing), not just a chart. Bounded and reuses the existing OIDC/cosign publish pattern, but it's net-new CI surface — call it out as part of the cost.
  2. Delivery-shape change for the pilot tenant: its artifact moves from raw manifests → a thin HelmRelease. Verify prune / release-ownership semantics on removal differ acceptably from a raw Kustomization (test in the pilot).
  3. ksail workload validate builds each kustomization standalone → needs --skip-kinds for HTTPScaledObject (and any CRD-backed instance) when scale-to-zero is templated, same as existing CRD-backed apps.

Recommendation

A platform-published, cosign-signed webapp OCI application chart, referenced by a thin per-tenant HelmRelease that lives in the tenant artifact. Adds no controller to the fragile layer, gives DRY with central evolution, keeps the tenant-owned-artifact boundary intact, and reuses the Tenant-archetype Helm decision (#1932) so there's one archetype mechanism, not two. KRO deferred (post-1.0). This is consistent with #1932's closing note: a Tenant hosts WebApp workloads, so WebApp is the same Flux→HelmRelease model, one layer down.

Consequences / pilot (remaining acceptance criteria)

1. Build the webapp application chart: Deployment+Service, HTTPRoute (central-Gateway parentRef), conditional KEDA 5-point set behind values.scaleToZero, conditional ExternalSecret behind values.secretRefs; values.schema.json for light typing; chart templates the Flux-best-practice HelmRelease fields.
2. Add a publish-chart workflow (OIDC + cosign) publishing to ghcr.io/devantler-tech/charts/webapp.
3. Pilot-migrate ascoachingogvaner (no SOPS, no scale-to-zero — the simplest case) local-first: tenant artifact emits a thin HelmRelease; render-diff vs current manifests; confirm ksail workload validate parity.
4. Second migration wedding-app exercises the SOPS-secret + (optional) scale-to-zero conditionals, including the platform-side ReferenceGrant addition if scale-to-zero is wanted.
5. Document "add a webapp" as the ~6-line values block in docs/TENANTS.md.

Sequencing with #1932

The Tenant chart (registration skeleton, in-repo) and the WebApp chart (workload, published OCI) are independent and complementary — either can pilot first. Suggested order: land the Tenant chart first (it's lower-risk, in-repo, no new publish pipeline), then WebApp, since WebApp's only net-new infrastructure is the chart-publish workflow.

If you'd rather record this as a committed design doc (e.g. docs/webapp-archetype.md, matching the topic-doc style — there's still no ADR dir, docs/dr is Disaster Recovery), say the word and I'll open a draft PR. Build-vs-buy and the publish-pipeline cost are your calls; this is the reasoned recommendation to act on.

[devantler]

🤖 Generated by the Daily AI Assistant

ADR update — WebApp archetype: Helm ruled out → CR-based (KRO)

Maintainer decision (direct): "Custom Resource approach — more cloud-native, so Helm charts are ruled out." This supersedes the Helm-chart recommendation in my prior comment here.

The engine decision is shared with the Tenant archetype (#1932) — one mechanism, not two. Full evaluation (Crossplane vs KRO vs extending ksail-operator, grounded in what's deployed on main) is in the revised ADR on #1932. Summary as it applies to WebApp:

• Crossplane — rejected for this: deployed but prod-only + only provider-upjet-github (external-API role) + no provider-kubernetes; composing in-cluster Deployment/Service/HTTPRoute via Crossplane Object wrappers is off-label. Keep Crossplane for the GitHub/Hetzner external APIs.
• KRO ResourceGraphDefinition — recommended: a typed, namespaced WebApp CRD whose instance renders the children directly; includeWhen is a clean fit for this archetype's optional parts:
  • HTTPRoute → always, parentRefs:[{name: platform, namespace: kube-system, sectionName: https}] (the central Cilium Gateway, verified); app supplies only host.
  • includeWhen scaleToZero → the verified KEDA 5-point recipe: initial replicas, HTTPScaledObject, HTTPRoute backend repointed to keda-add-ons-http-interceptor-proxy.keda.svc:8080, CiliumNetworkPolicy ingress from keda, drain-safe PDB. Caveat unchanged: the cross-ns ReferenceGrant (.../keda-http-add-on/reference-grant.yaml) enumerates app namespaces → a new tenant's scale-to-zero still needs a one-line platform-side ReferenceGrant addition (can't come from the tenant artifact).
  • includeWhen secretRefs → ExternalSecret (ESO + SOPS for wedding-app); never inlined.
  • Kyverno mutate still owns securityContext/anti-affinity/topology-spread — the RGD must not hardcode conflicting values.
• ksail-operator ownership is the maximal-ownership alternative (own the WebApp CRD in your operator vs adopt KRO) — see Generalize the Tenant abstraction — evaluate Capsule vs KRO vs Helm library chart #1932; your B-vs-C call.

Placement (the WebApp-specific open question)

The WebApp CRD/RGD is platform-defined (KRO RGD lives in the platform repo, in the shared layer); a tenant's OCI artifact carries only the small WebApp instance (the typed CR). Platform owns the schema/rendering, tenant owns the values — same split as before, now via a CR instead of a chart's values. Works because KRO is in the shared layer (unlike prod-only Crossplane).

Pilot

Same as #1932's plan: ascoachingogvaner first (no SOPS/scale-to-zero) — render-diff the WebApp RGD output vs current manifests, confirm ksail workload validate parity (add the CRD to skipKinds); then wedding-app exercises the SOPS + scale-to-zero conditionals + the ReferenceGrant step.

I'll open the draft PR (KRO + Tenant/WebApp RGDs + local-first pilot) once you confirm KRO vs ksail-operator on #1932.

[devantler]

🤖 Generated by the Daily AI Assistant

Decision (maintainer, 2026-06-19): KRO.

The WebApp archetype is settled on KRO (kro.run ResourceGraphDefinition), consistent with the Tenant decision in #1932 (Helm ruled out; Crossplane reserved for external-API/prod-only).

Sequencing: this builds on the KRO install + Tenant RGD groundwork in #1932. The WebApp RGD composes the per-app Deployment/Service/HTTPRoute/(optional KEDA scale-to-zero + ExternalSecret) skeleton, with optional children via includeWhen. Pilot local-first after the Tenant RGD lands.