Skip to content

Fix md5srv HIL job failing with HTTP 403 while installing Bats - #257

Closed
zakoverflow wants to merge 2 commits into
blues:masterfrom
zakoverflow:fix/hil-bats-action-403
Closed

zakoverflow wants to merge 2 commits into
blues:masterfrom
zakoverflow:fix/hil-bats-action-403

Conversation

@zakoverflow

Copy link
Copy Markdown
Contributor

Problem

The nightly note-c card.binary HIL tests run on 2026-10-08 failed in its first job,
md5srv-test / test-md5srv, at Setup Bats and bats libs. Because
notecard-binary-test has needs: md5srv-test, the hardware job was skipped.

Run: https://github.com/blues/note-c/actions/runs/37747611386

Run bats-core/bats-action@3.0.0
  with:
    bats-install: true          # note: no bats-version input
...
Download and install Bats
  VERSION=
  if [[ -z "$VERSION" ]] || [[ "$VERSION" = "latest" ]]; then
    VERSION=$(curl -fsSL --retry 4 --retry-connrefused https://api.github.com/repos/bats-core/bats-core/releases/latest | ...)
curl: (22) The requested URL returned error: 403
##[error]Process completed with exit code 1.

Root cause

  • md5srv-tests.yml never passed bats-version to bats-core/bats-action. The
    BATS_VERSION: 1.10.0 env var in that workflow is not an action input, so the action
    never read it. Every run looked up the "latest" release through the GitHub REST API at
    api.github.com/.../releases/latest. The 2026-10-07 run, for example, installed
    Bats v1.14.0.
  • bats-action@3.0.0 makes that API call without a token. Unauthenticated REST calls share
    a 60 requests/hour budget per source IP, and hosted runners share IPs. curl -f drops
    the response body, so the log can't prove the 403 was a rate limit. It's the most likely
    cause.

Change

.github/workflows/md5srv-tests.yml only:

     env:
         TERM: xterm-256color
         MD5SRV_TIMEOUT: 5
         MD5SRV_DIR: ./test/hitl/scripts
-        BATS_VERSION: 1.10.0
     steps:
       - name: Setup Bats and bats libs
         id: setup-bats
-        uses: bats-core/bats-action@3.0.0
+        uses: bats-core/bats-action@3.0.1
         with:
             bats-install: true
+            bats-version: 1.14.0
             file-install: false
             detik-install: false
  • Pin Bats to 1.14.0. With an explicit bats-version, the action skips the
    releases/latest lookup entirely and fetches only the tag tarball. So the fix no longer
    depends on the rate-limit theory: the failing call is simply never made. 1.14.0 is the
    version the nightly has been installing as "latest", so the tests run on the same Bats
    as before.
  • Remove BATS_VERSION: 1.10.0. Nothing read it, and its value didn't match the Bats
    version actually in use.
  • Bump bats-action 3.0.0 → 3.0.1 (patch release; upstream pin github action hash and add scorecard workflow and fix the issue with latest version bats-core/bats-action#24).
    3.0.1 sends the job's GITHUB_TOKEN with the Bats tarball download (and with the
    releases/latest lookup, which the pin now skips). That token stays as defense in depth
    for the tarball download. The bats-support and bats-assert tarballs are still fetched
    without a token, the same as before. The tag pin follows this repo's existing convention.

Verification

  • actionlint 1.7.12 reports no errors on md5srv-tests.yml. With shellcheck enabled,
    the only finding is an info-level SC2086 on the existing, unchanged "Re-run Tests"
    step. It shows up identically on master.
  • I ran this branch's md5srv-tests.yml unchanged in a scratch push-triggered workflow
    on my fork. It used a fresh branch with no cache entry for Linux-X64-bats-1.14.0, so
    the install really ran. No secrets, no hardware, and no Notehub or Notestation access.
    https://github.com/zakoverflow/note-c/actions/runs/37817746692
    • with: bats-version: 1.14.0 → VERSION=1.14.0, so the releases/latest branch is
      skipped.
    • Cache not found for input keys: Linux-X64-bats-1.14.0
    • Bats v1.14.0 installed in /home/runner/.local/share/bats
    • 14 tests, 0 failures
    • A sibling job in the same run used the same action inputs with a logging curl shim.
      The only URLs requested were the three tag tarballs (bats-core/.../v1.14.0.tar.gz,
      bats-support/.../v0.3.0.tar.gz, bats-assert/.../v2.1.0.tar.gz). There was no
      api.github.com call.

Not verified here: a real run on blues/note-c. A workflow_dispatch or the next nightly
is the final check for the md5srv-test job.

Known follow-up (not fixed by this PR)

  • The bats-action cache often reports Cache hit for: <key> and then
    Failed to restore: Cache not found for input keys: <key>. On blues/note-c this happens
    every night for Linux-X64-bats-, so the install runs every time. I saw the same pattern
    on my fork (Linux-X64-bats-assert-2.1.0 in the run above), so it doesn't look specific
    to this repo's cache. Cause unknown. With the pin, a failed restore only means an extra
    tarball download.

- The md5srv-test job (the first job of the nightly card.binary HIL
  workflow) failed in "Setup Bats and bats libs" with
  "curl: (22) The requested URL returned error: 403". The card.binary
  hardware job needs md5srv-test, so it was skipped.
- The workflow does not pass bats-version to bats-core/bats-action, so
  the action looks up the latest Bats release through
  api.github.com/repos/bats-core/bats-core/releases/latest. The
  BATS_VERSION env var in this workflow is not read by the action.
- bats-action 3.0.0 makes that lookup without a token. Unauthenticated
  API calls share a 60 requests/hour limit per source IP, and GitHub
  answers with 403 once the limit is used up. curl -f hides the response
  body, so the log can't confirm the reason, but this is the likely one.
- bats-action 3.0.1 sends the job's GITHUB_TOKEN with the lookup and the
  tarball download. That is the upstream fix for this lookup. Bump to
  3.0.1. No other change: same inputs, same Bats version resolution, same
  libraries.
- Checked locally: actionlint reports no issues in md5srv-tests.yml, and
  the 3.0.1 install logic resolves and installs Bats v1.14.0, after which
  the md5srv bats suite passes (14 tests, 0 failures).
- Pass bats-version: 1.14.0 to bats-core/bats-action. With an explicit
  version the action skips its api.github.com releases/latest lookup, so
  the install no longer depends on GitHub API rate limits at all.
- 1.14.0 is the version the nightly has been installing as "latest", so
  this does not change which Bats the tests run on.
- Remove the BATS_VERSION: 1.10.0 env var. The action never read it, and
  its value did not match the Bats version actually in use.
- Keep bats-action 3.0.1. Its token on the tarball download stays as
  defense in depth.
@zakoverflow

Copy link
Copy Markdown
Contributor Author

Dev Lead review (sign-off) for dadeb41.

  • Scope: two-line change to .github/workflows/md5srv-tests.yml. It matches the stated root cause, and nothing unrelated is touched.
  • Design: pinning bats-version: 1.14.0 removes the releases/latest lookup entirely, so the job no longer depends on the unauthenticated API budget, and nightly runs stop drifting to new Bats releases. Removing the unused BATS_VERSION: 1.10.0 env var takes out misleading dead state. bats-action@3.0.1 adds the job token as defense in depth for the bats-core tarball download.
  • Evidence: all PR checks are green, including md5srv-test / test-md5srv. A fork run (37817746692) shows VERSION=1.14.0, no api.github.com call, and md5srv passing 14/14.
  • Follow-ups (not blocking): the bats cache fails to restore every night, and bats-support and bats-assert are still fetched without a token.

Signed off. Ready for a maintainer to merge. notecard-binary-test is skipped on PRs; the HIL station outage it hit is tracked separately and has since been resolved on the lab side.

@zakoverflow

Copy link
Copy Markdown
Contributor Author

Closing as superseded. Upstream fixed the same Bats install failure in #259 (2d22844), which uses bats-core/bats-action@4.0.0 with Bats 1.14.0. A rebase would leave this branch with no diff. Thank you.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant