Skip to content

notify.md

Latest commit

 

History

History
221 lines (176 loc) · 12.2 KB

notify.md

File metadata and controls

221 lines (176 loc) · 12.2 KB
Raw

notify-failure.yml: Slack incident threads

The reusable workflow .github/workflows/notify-failure.yml keeps one Slack thread per CI incident. The logic is cmd/notify, with the rules in internal/notify; the workflow builds it from this repo at the commit it is called at (job.workflow_sha, the caller's @<sha>).

Which runs

A run is tracked when:

  • (a) it's for a pull request opened by ccf-release-bot[bot], or for a branch starting with renovate/ or ccf-bump/ (a pull request, or a push to that branch);
  • (b) it's for a push to the default branch.

Every other run (a human's PR, a push to another branch, a tag, a schedule) does nothing. A run for a closed pull request, merged or not (the event payload's pull_request.state is closed), runs no failure logic whatever its result: it opens no incident, posts no failure or recovery and is never posted as needing a human (release-please relabels its PR after the merge, and that re-runs CI on a PR no one can act on). For a PR that rule (a) tracked, it only closes what is still open, see Closed PRs.

Incidents

An incident is keyed by the repo + the caller workflow's file + the pull request number for pull_request runs, or + the branch for pushes. Each workflow file has its own incidents, so two workflows of one repo don't close each other's.

The run's result comes from the needs input, the calling job's toJSON(needs): any job with result failure makes it a failure, listing those jobs; otherwise any cancelled job, or no job having run, leaves the incident as it is; otherwise (successes, with or without skips) it's a pass.

Result Incident Posts Then
failure none, or closed a top-level card in the channel (Failing) open, with the card's ts
failure open, same commit and failed jobs nothing (a re-run) unchanged
failure open, another commit or other failed jobs a reply in the thread, and the card edited to Failing again open
pass open ✅ passing again in the thread, and the card edited to Resolved closed: the next failure starts a new card
pass none, or closed nothing unchanged
cancelled / all skipped any nothing unchanged

The image check

ci-go-service.yml, ci-ui.yml and ci-action.yml run ci-image.yml as job image beside required, not under it (image-scan.md): it gates no merge, but a red image still fails the caller's ci job. So these workflows report their required check's outcome as the required-result output, which the caller's toJSON(needs) carries ({"ci": {"result": "failure", "outputs": {"required-result": "success"}}}). A job that failed although its required check passed failed in image alone, and notify reads it as:

  • for the run's incident, a pass: on a pull request (Renovate, ccf-bump, release-please, image-scan/os-patches) or an automation branch, a failure of image alone opens nothing and posts nothing, since it doesn't block the PR;
  • on a push to the default branch, a failure of ci / image in an incident of its own: a red image on main is the signal that an image needs a fix. It follows the same table (open, reply, recover with ✅ passing again once image passes), with its own card listing ci / image, so a build or test failure on main while it is open still opens or replies in the run's incident. When the required check fails, image's own result is unknown, and the image incident is left as it is until a run whose required check passes.

A job without the output (another ci-*.yml, or a run whose required job didn't run) counts as before, by its own result.

The top-level message is a Block Kit card (slackkit.IncidentCard, slack.md): a red bar while failing and green once resolved; the header CI failing · <repo> (or failing again, resolved); the fully qualified PR (owner/repo#N, linked) and its title, or owner/repo@<branch> for a push; fields Status, Workflow, Opened at (the commit the incident opened at, and when), Resolved (when, and after how long) and Failed jobs (the latest failure's, struck through once resolved); buttons View PR and Latest run. Every transition replies in the thread first and then edits the card with chat.update. If the edit fails (say, the message was deleted), the job logs a warning and still saves the state, so the reply is not posted twice.

Thread replies (Slack mrkdwn; the short SHA links to the commit, run to the run):

❌ failed: ci at 3333333 · run
✅ passing again at 4444444 · run

The commit is the PR head for pull requests (not the test merge commit), so re-runs after the base moves count as the same commit.

State

The state of an incident (channel, the card's ts, open, the last posted commit and failed jobs, and for the card opened_sha, opened_at, again and resolved_at) is a small JSON file kept in the Actions cache. A state saved before the cards lacks the card fields, and its card shows the last commit instead and no times:

  • the key is ccf-notify-incident-<hash of the incident key>-<run id>-<run attempt>, so every save is a new entry; the image check's incident is ccf-notify-image-incident-<hash>-... in its own file, a prefix that neither restores the other's state;
  • the job restores with that prefix as restore-keys, and the cache returns the newest entry with it, so the latest state wins;
  • it saves only when it posted (so a pass with no incident, or a re-run, saves nothing).

Cache scoping does the rest: a pull_request run saves under the PR's merge ref, which later runs of the same PR read; a push to main saves under main, which later main runs read. A PR's key never matches main's, so PRs don't pick up main's incident.

Replies go to the incident's own channel; the channel input only picks where new incidents start. No GitHub permission beyond contents: read and no Slack scope beyond chat:write (replies are chat.postMessage with thread_ts; the bot edits its own cards with chat.update).

Limits:

  • Caches unused for 7 days are evicted. An incident with no run for a week loses its state, and its next failure starts a new thread (its next pass posts nothing).
  • Two runs of the same incident finishing at the same moment both read the same state, so both may open a thread. CI workflows that cancel superseded PR runs make this rare.
  • A state that can't be read (bad JSON, another key) is reported in the log and treated as no incident.

Needs a human

A second rule, separate from incidents, posts a tracked bot PR that waits for a person to the channel in the SLACK_CHANNEL_NEEDS_HUMAN variable, once per PR (see attention.md). With the variable empty the rule is off: nothing is posted, the incidents work as before, and nothing fails.

For a pull_request run tracked by rule (a) (notify.Route):

The PR Posts to SLACK_CHANNEL_NEEDS_HUMAN Incident
carries the needs-human label (in the event payload) Why: Major update (renovate/*), OPA update in the api (never auto-merged) (renovate/opa), Auto-merge off (...) (ccf-bump/*), or Labelled needs-human as usual
a release-please PR (release-please--*) whose release-checks job failed Why: Release PR blocked by release-checks (e.g. needs release:major-approved, or an internal dep isn't final) none when release-checks is the only failed job (the incident sees the run as cancelled and stays as it is); as usual when other jobs failed too
anything else, or a human's PR nothing as usual

The post is a card (slackkit.NeedsHumanCard, amber bar): the PR (owner/repo#N, linked) and its title, fields Why, CI (the run's result: passing, or the failed jobs), Opened by and Waiting (since the PR was opened), and a Review PR button. The record saved for it keeps the card's channel, ts, reason and CI, to edit the card later.

Once per PR: the job restores the cache key ccf-notify-needs-human-<hash of repo + PR>, posts only on a miss, and then saves a small record under that key. Re-runs, pushes and other workflows of the same PR find it and post nothing. Like incident state, the record is evicted after 7 days without use, so a PR idle for a week may be posted again; two runs of one PR finishing at the same moment may both post. The weekly attention digest lists every PR still waiting either way.

The callers must run CI on labeled (pull_request types: [opened, edited, synchronize, reopened, labeled, unlabeled, closed], see Calling it), because Renovate and ccf-bump add the label after opening the PR: the labeled run is the one that sees it. The job's needs must include the release-checks job under that name.

Closed PRs

Callers add closed to their pull_request types (below). The CI jobs of the kind workflows skip on a closed PR, so only this job does anything, and for a PR that rule (a) tracked it runs notify closed instead of the failure logic:

Open item Becomes
the PR's needs-human card edited to Handled (green when merged, grey when closed): "Merged by " or "Closed by ", and how long it waited
an open incident on the PR a reply ⚪ PR merged by <actor>; incident closed in its thread, the card edited to "Closed (PR merged)" or "Closed (PR closed)" (grey bar, failed jobs struck through), and the incident closed

The actor is the PR's merged_by, or the event's sender for a PR closed without merging. Nothing open, nothing posted. Card edits that fail are warnings, as for incidents; a reply that fails fails the job. A closed PR whose needs-human record was evicted (7 days idle) keeps its card as it was.

Calling it

# in the consuming repo's ci.yml
on:
  pull_request:
    # labeled: the needs-human label; closed: mark the PR's cards handled / closed.
    types: [opened, edited, synchronize, reopened, labeled, unlabeled, closed]
# ... the CI jobs, then:
  notify:
    needs: [ci, release-checks]  # every job whose failure should notify
    if: always()
    uses: compliance-framework/workflows/.github/workflows/notify-failure.yml@<sha> # vX.Y.Z
    with:
      needs: ${{ toJSON(needs) }}
    permissions:
      contents: read
    secrets: inherit
Input Default What
needs "" The calling job's toJSON(needs): the jobs and their results. Empty means the run failed, with the failed jobs unknown (callers still on if: failure()).
channel "" Slack channel ID for new incidents; empty means the SLACK_CHANNEL_CI_FAILURES variable.
workflows-ref "" Ref of this repo to build cmd/notify from, to test another ref; empty means the commit the workflow is called at. Callers don't pass it: ccf-bump deletes it when it moves the job's pin.

Secret: SLACK_BOT_TOKEN (optional, chat:write), via secrets: inherit. Without it (forks, repos outside the secret's scope) the job does nothing and succeeds. Variables: SLACK_CHANNEL_CI_FAILURES (incidents, unless channel is given) and SLACK_CHANNEL_NEEDS_HUMAN (optional, Needs a human).

The job now runs after every CI run, so passes also pay for building cmd/notify; runs that aren't tracked stop right after the build.

Migrating from if: failure()

The earlier caller ran the job only on failure, with no inputs, and granted actions: read:

  notify:
    needs: [ci]
    if: failure()
    uses: compliance-framework/workflows/.github/workflows/notify-failure.yml@<sha> # vX.Y.Z
    permissions:
      actions: read
      contents: read
    secrets: inherit

Change if: failure() to if: always(), add with: {needs: ${{ toJSON(needs) }}}, and drop actions: read (no longer used; keeping it is harmless). Until a repo migrates, its failures still open and reply in threads, without the failed job names, but its incidents never close: the job doesn't run on passes, so nothing posts the recovery.

What changed for the default branch: it used to post only when the previous push run had passed (one message when main broke). Now the open incident does that job: the first failure opens a thread, each later failing commit replies in it, and the first pass closes it. The old ccf-notify-failure-* dedupe cache entries are no longer read.