Skip to content

feat(ssp-templates): discovery worker and periodic job - #525

Merged
gusfcarvalho merged 2 commits into
feat/ssp-templates-3-plannerfrom
feat/ssp-templates-4-discovery
Oct 22, 2026
Merged

gusfcarvalho merged 2 commits into
feat/ssp-templates-3-plannerfrom
feat/ssp-templates-4-discovery

Conversation

@ccf-lisa

@ccf-lisa ccf-lisa Bot commented Oct 19, 2026 •

Copy link
Copy Markdown
Contributor

Place in the stack

SSP templates (feature 2 of ssp-templates.md), PR B4 of 5. Base: B3 (#524, feat/ssp-templates-3-planner). Below it: B2 (#523), B1 (#522) and the SSP scope stack (#516–#520). The design's B4 is split in two to stay under 1000 lines. This PR is discovery. B5 (feat/ssp-templates-5-routes) adds the run route, the instances list (stale, key-hash, skipped-entries) and the forget route.

What it adds

  • ssptemplates.Discovery runs the B3 planner against the latest evidence and applies it:
    • New group: creates the SSP through B2's sspservice in a transaction holding the template's advisory lock, records the instance, and learns the scope (sspscope.Service.Learn, origin template).

      • The SSP id is uuid.NewSHA1(namespace, templateID || keyHash).
      • metadata.title is the rendered title and metadata.version is 1.0.
      • Props are the rendered props plus ssp-template=<template id>.
      • system-name is the title, the description is rendered (or falls back to the title), and status is under-development.

      Profiles are attached after that transaction commits. A failed attach is recorded in last_error and profiles_pending, and the next run retries it.

    • Existing group: refreshes last_seen_at and learns new scope entries. Learn already skips exceptions and existing entries.

    • Deleted SSP (instance ssp_id null): left alone, never recreated.

    • At most MaxNewSSPsPerRun (100) new SSPs per template per run; the next run creates the rest.

    • Learned entries the scope cannot hold (from the planner, and LearnResult.Skipped) go into new instance columns skipped_entries and skipped_sample (up to 5).

    • A failing group gets last_error and the run's error; the other groups still apply. last_run_at is set when the run finishes. This is the contract addition from review: B1 exposes it as last-run-at, null until this job sets it.

  • River worker and periodic job ssp_template_discovery on the scheduler queue. Empty args run every enabled template, loading the evidence once; args with template_id run one template, for B5's run route.
  • The worker reuses the ProfileControlResolver that cmd/run.go already injects (the oscal handler's control resolution), adapted to sspservice.ControlResolver.
  • Config: CCF_SSP_TEMPLATE_DISCOVERY_SCHEDULE (default hourly, 0 37 * * * *) and CCF_SSP_TEMPLATE_STALE_AFTER (default 168h, used by B5), in internal/config and .env.example.

How it was tested

  • Integration (ssptemplates/discovery_integration_test.go, testcontainers):
    • the SSP is created with the evidence's casing, the deterministic id, title, description, props, the template's profile and its implemented requirements; scope entries have origin template and the template id; last_run_at is set;
    • runs are idempotent (one SSP, last_seen_at refreshed);
    • later runs learn new values (evidence in other casing joins the same SSP) and respect exceptions;
    • a deleted SSP is never recreated;
    • the 100-per-run cap: 101 groups give 100, then 1;
    • a failed profile attach is recorded and then retried successfully;
    • disabled templates are skipped;
    • invalid learned entries are counted and sampled.
  • Unit: the config loader, the schedule fallback, and the resolver adapter's case-insensitive de-duplication.
  • Gate on this branch:
    • make swag (no doc change), make lint (only the existing SA1019), make test, go vet -tags integration ./...;
    • integration suites of internal/service/worker, ssptemplates, internal/service, internal/service/relational, internal/config;
    • make check-diff.

Changed lines excluding generated swagger: 762.

Deviations from design

  • The design's B4 is split into B4 (discovery) and B5 (routes).
  • New instance columns, not in the design's model:
    • profiles_pending: retry an attach only while one failed, so a profile a user removed is never re-attached;
    • skipped_entries / skipped_sample: the design's "skipped count and a sample".
  • The ssp-template prop uses the existing relational.CCFOSCALNamespace (https://compliance-framework.github.io/ns/oscal), not …/ns.
  • Rendered prop values that come out empty are dropped, because OSCAL props need a value.
  • plan.InvalidEntry is now an alias of relational.SkippedScopeEntry.

Open questions for reviewers

  • An existing SSP whose evidence drops under min-streams is not refreshed, so it can be reported stale even though some evidence remains. Should min-streams apply only to creating SSPs?
  • The ssp-template prop namespace: the existing …/ns/oscal constant, or the design's …/ns?

🤖 Generated with Claude Code

@coderabbitai

coderabbitai Bot commented Oct 19, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration
  • Configuration used: Organization UI
  • Review profile: ASSERTIVE
  • Plan: Advanced
  • Run ID: 9cea4713-11ca-4840-a96e-62c29145911e

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

ccf-lisa Bot and others added 2 commits October 19, 2026 11:49
Apply SSP templates to the latest evidence: create one SSP per group
(deterministic id, rendered title/description/props, the template's
profiles attached), record it as an instance, and learn the group's scope.
A periodic River job (CCF_SSP_TEMPLATE_DISCOVERY_SCHEDULE) runs every
enabled template; CCF_SSP_TEMPLATE_STALE_AFTER configures stale reporting.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ccf-lisa
ccf-lisa Bot force-pushed the feat/ssp-templates-4-discovery branch from daa9052 to cf52115 Compare October 19, 2026 14:53

@gusfcarvalho gusfcarvalho left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ccf-review: APPROVE

no findings.

Stack (gh stack 521): #516 → #517 → #518 → #519 → #520 → #522 → #523 → #524 → #525 → #526

@gusfcarvalho
gusfcarvalho merged commit c7856f1 into main Oct 22, 2026
34 of 37 checks passed
@gusfcarvalho
gusfcarvalho deleted the feat/ssp-templates-4-discovery branch October 22, 2026 08:44
gusfcarvalho pushed a commit that referenced this pull request Oct 22, 2026
## Place in the stack

SSP templates (feature 2 of `ssp-templates.md`), PR **B1 of 5** (B1–B5:
#522, #523, #524, #525, #526), stacked on the SSP scope stack
(#516–#520, top `feat/ssp-scope-5-auto-filters`).

## What it adds

- Models `SSPTemplate` and `SSPTemplateInstance`
(`internal/service/relational/ssp_templates.go`), in both AutoMigrate
lists and both drop lists. Their foreign keys (instance → template `ON
DELETE CASCADE`, instance → SSP `ON DELETE SET NULL`) are added by
`EnsureSSPConstraints`, which is `EnsureSSPScopeConstraints` renamed,
with two keys appended. AutoMigrate here does not enforce gorm FK tags.
- Package `internal/service/relational/ssptemplates`:
- `Build`: validation and normalization. Keys are trimmed, lower-cased
and sorted (1–5). Scope-label groups are normalized the same way (≤10
groups of 1–5 labels). Keys and labels follow `sspscope.Normalize`'s key
rules, so anything a template accepts can be stored as a scope entry.
`min-streams` must be ≥ 1. The title template is required, and the
title, description and prop templates must parse. The labelfilter must
be valid.
- `Service`: list/get/create/update/delete. Profile ids must exist. Keys
cannot change on update (400).
- Admin routes `/api/admin/ssp-templates` (list, create, get, update,
delete), behind `JWTMiddleware` + `pep.Authorize(authz.ResourceAdmin,
authz.ActionManage)`, like `/admin/subject-templates`. JSON is
kebab-case. Swagger is regenerated.
- **Contract addition from review**: template responses also return
`ssp-count` and `last-run-at`.
  - `ssp-count` is the number of instances whose SSP still exists.
- `last-run-at` is a new nullable `last_run_at` column. It stays `null`
until B4's discovery job sets it, and Update preserves it.

Not in this PR: preview (B3), SSP creation (B2/B4), discovery, run,
instances routes (B4).

## How it was tested

- Unit: `ssptemplates/validate_test.go`. It covers normalization,
defaults, and every rejection rule.
- Integration
(`internal/api/handler/templates/ssp_template_integration_test.go`) runs
under the embedded Cedar PDP:
  - an admin creates, lists, gets, updates and deletes templates;
  - changing keys is rejected;
  - a contributor gets 403 on every route;
  - 400/404 cases;
- the FK behaviour and `ssp-count`: deleting an SSP nulls its instance's
`ssp_id` and drops it from `ssp-count`, and deleting a template drops
its instances and keeps its SSPs.
- Gate on this branch:
- `make swag`, `make lint` (only the existing SA1019 in
`internal/api/metrics.go`), `make test`, `go vet -tags integration
./...`;
- integration suites of `internal/api/handler`,
`internal/api/handler/templates`, `internal/service`,
`internal/service/relational`, `internal/service/relational/sspscope`;
  - `make check-diff`.

Changed lines excluding generated swagger: 1000.

## Deviations from design

- The instance model has exactly the design's fields. The
`skipped-entries` count and sample reported by the instances route are
added in B4, with the code that writes them.
- Duplicate scope-label groups (after normalization) are rejected with
400, not merged silently. Duplicate profile ids are de-duplicated.
- An empty filter (`{}` / `{"scope":{}}`) matches everything, so it is
stored as no filter.
- The `enabled` column has no database default (the design has
`default:true`). gorm writes a column default over a false `Enabled` on
create, so a disabled template was stored as enabled. `Build` always
sets `Enabled`, with nil meaning true.
- The list route is paginated (`svc.ListResponse`, `page`/`limit`) like
subject templates.
- The response timestamps are kebab-case (`created-at`/`updated-at`),
unlike subject templates' camelCase ones.

## Open questions for reviewers

- Templates are only checked to parse. Should save also reject templates
that reference a field that is not one of the keys (keys are
lower-cased, so `{{ ._APP }}` would render empty)? The design asks only
for parse errors.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: ccf-lisa[bot] <286799724+ccf-lisa[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
gusfcarvalho pushed a commit that referenced this pull request Oct 22, 2026
## Place in the stack

SSP templates (feature 2 of `ssp-templates.md`), PR **B5 of 5**. It is
the second half of the design's B4, split to keep each PR under 1000
lines. Base: B4 (#525, `feat/ssp-templates-4-discovery`). Below it: B3
(#524), B2 (#523), B1 (#522) and the SSP scope stack (#516–#520).

## What it adds

All three routes sit behind the existing `/api/admin/ssp-templates`
group: JWT + `admin:manage`.

- `POST /api/admin/ssp-templates/{id}/run` queues a discovery run of the
template (`ssp_template_discovery` with the template id as args) and
answers **202**.
  - 404 for an unknown template, **409** for a disabled one.
  - 503 when no job queue is wired, and 500 when the worker is disabled.
- Runs are unique by args among waiting or running jobs, so a finished
run never blocks the next one.
- `worker.Service.EnqueueSSPTemplateDiscovery` implements the new
`templates.SSPTemplateRunEnqueuer`, wired through
`APIServices.SSPTemplateRunEnqueuer` in `cmd/run.go`.
- `GET /api/admin/ssp-templates/{id}/ssps` is paginated and newest
first. Each item has:
- `key-values` and **`key-hash`** (a contract addition from review: the
forget route's id);
- `ssp-id` (null once the SSP was deleted) and the SSP's current
`title`;
  - `created-at` and `last-seen-at`;
- `stale`: the SSP exists and was last seen more than
`CCF_SSP_TEMPLATE_STALE_AFTER` ago (from B4's config);
  - `last-error`, `skipped-entries` and `skipped-sample`.
- `DELETE /api/admin/ssp-templates/{id}/ssps/{keyHash}` forgets an
instance whose SSP was deleted, so the next run creates it again. It
answers 409 while the SSP exists and 404 for an unknown instance.

Swagger is regenerated.

## How it was tested

- Integration under the embedded Cedar PDP
(`ssp_template_instances_integration_test.go`):
- run: queued once, disabled 409, unknown 404, contributor 403, no job
queue 503;
- list: key values, key hash, title, stale/fresh/deleted, last error,
skipped entries; contributor 403; unknown template 404;
- forget: 409 while the SSP exists, 404 unknown, contributor 403, 204
for a deleted SSP, and the list shrinks afterwards.
- Unit: enqueueing with the worker disabled is an error.
- Gate on this branch:
- `make swag`, `make lint` (only the existing SA1019), `make test`, `go
vet -tags integration ./...`;
- integration suites of `internal/api/handler`,
`internal/api/handler/templates`, `ssptemplates`,
`internal/service/worker`;
  - `make check-diff`.

Changed lines excluding generated swagger: 395.

## Deviations from design

- This PR exists only because B4 was split.
- The instance list is paginated, and adds `title`, `key-hash` (contract
addition) and `skipped-sample` to the design's fields.
- A run requested while the worker is disabled answers 500 instead of a
202 for a run that would never happen.

## Open questions for reviewers

- Should a run requested while the worker is disabled answer 503 rather
than 500? Telling that case apart from an enqueue failure needs an error
exported from the worker package.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: ccf-lisa[bot] <286799724+ccf-lisa[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant