Production deploy stuck on 2.5.0 — grabthar working tree not updating; build logs inaccessible

[jonfroehlich]

Summary

Production (makeabilitylab.cs.washington.edu) is stuck on release 2.5.0 and will not advance, despite multiple tag pushes (2.8.2, 2.8.2.1). The deploy pipeline fires and rebuilds/restarts the container, but it keeps running old code because the git working tree on the prod host (grabthar) is not being updated to the new commit before the Docker build runs. Separately, we can no longer access the build logs (buildlog.txt) to debug this directly.

This was discovered while shipping the sitemap work (#1252, PRs #1307 / #1308 / #1310), which is fully validated on the test server but cannot reach production.

Symptoms

• Prod admin shows 2.5.0, even though master is at 2.8.2 and tags 2.8.2/2.8.2.1 were pushed.
• GET /sitemap.xml on prod → 404 (the route exists only in the new code).
• GET /robots.txt on prod → still the old body (no Sitemap: line), even though the committed ./robots.txt changed several pushes ago.
• After each tag push, prod briefly 502s and restarts, then comes back still on 2.5.0.

Diagnosis: the prod working tree is not advancing

The deploy is running (the container rebuilds and restarts), but it builds stale source:

• debug.log shows runserver restarting at the deploy times (e.g. Watching for file changes with StatReloader at 15:34 and 15:58), so docker compose up is recreating the container.
• The most recent build did a fresh, uncached COPY . /code/ (BuildKit, image sha256:c4af85f2…) — and prod still served the old release. So the build copied old source from grabthar's disk.
• The static ./robots.txt is served by Apache straight from the checkout (no Docker involved). The committed file changed in PR Serve sitemap via static robots.txt; drop dead Django robots view (#1252) #1308, yet prod's /robots.txt is unchanged. A plain file in the repo changed and never appeared on the prod disk → the working tree itself is not updating.

Ruled out

• ❌ Our code / the tag / the webhook firing — webhook deliveries succeed; code is verified on -test.
• ❌ Docker layer caching — the latest build re-ran COPY . /code/ uncached and still produced old behavior.
• ❌ A build failure — builds complete successfully (Successfully built … / exporting to image … done).
• ❌ Disk / dependency errors — no errors in the build output.

Most likely cause

The webhook's git fetch / git checkout step for the makeabilitylab.cs.washington.edu checkout on grabthar is not advancing the working tree — e.g. a detached HEAD, local modifications blocking a pull/checkout, or it's pinned to an old ref. Target commit: 0bc9dca (tag 2.8.2.1).

This requires CSE IT (Jason Howe) — we have no SSH/admin access to grabthar (see docs/DEPLOYMENT.md → server access model).

We can no longer access the build logs

Debugging the above is hampered because the build log is effectively inaccessible to maintainers:

• The documented web path /logs/buildlog.txt returns 403 Forbidden (and /logs/ is Shibboleth-gated / forbidden) on prod.
• buildlog.txt is not on the shared CSE filesystem — it is not under /cse/web/research/makelab/www/ on recycle (a find over the makelab tree turns up nothing). Only debug.log (+ rotated debug.log.1) is mounted out to the shared filesystem. (docs/DEPLOYMENT.md was corrected for this in docs: correct log locations in DEPLOYMENT.md (buildlog.txt not on shared FS) #1312.)
• The only reliable copy of the build output today is the deploy notification email sent on each push.

So when a prod deploy misbehaves, there's no self-serve way to read the build/deploy log — we have to rely on forwarded emails. We should ask CSE IT to either fix access to /logs/buildlog.txt or mount the build log to the shared filesystem (like debug.log).

Next steps

• CSE IT (Jason Howe): fix the git fetch/checkout step on grabthar so the prod working tree advances to the tagged commit (0bc9dca / 2.8.2.1).
• CSE IT: restore maintainer access to the build log — either un-forbid /logs/buildlog.txt or mount it to /cse/web/research/makelab/www/ alongside debug.log.
• Once prod advances: re-run the sitemap crawl validation against production (all <loc> → 200, https, correct domain), then submit the sitemap in Google Search Console.

References

• Sitemap work: Create sitemap for website? #1252 (PRs Add dynamic sitemap.xml and robots.txt (#1252) #1307, Serve sitemap via static robots.txt; drop dead Django robots view (#1252) #1308, Pin sitemap URLs to https; bump to 2.8.2 (#1252) #1310)
• Tags pushed: 2.8.2, 2.8.2.1 (commit 0bc9dca)
• Docs corrected: docs: document Apache static-vs-Django routing (robots.txt & sitemap.xml) #1311 (Apache routing), docs: correct log locations in DEPLOYMENT.md (buildlog.txt not on shared FS) #1312 (log locations)
• Server access model: docs/DEPLOYMENT.md

[jonfroehlich]

Resolved (for now): prod advanced to 2.9.0

CSE IT (Jason Howe) confirmed the root cause and manually advanced the prod working tree. Prod admin now shows 2.9.0.

Confirmed root cause — exactly the "local modifications blocking a pull/checkout" hypothesis

The deploy's git checkout was failing on a locally-modified, git-tracked robots.txt:

[root@grabthar makeabilitylabwebsite]# sudo -u apache git checkout tags/2.9.0
error: Your local changes to the following files would be overwritten by checkout:
    robots.txt
Please commit your changes or stash them before you switch branches.

The working tree could not advance because checkout refused to clobber a tracked file that had been edited on the server. Jason cleared it and did a manual pull/merge to land 2.9.0.

Why robots.txt was locally edited: prod and test need different crawler rules — prod allows indexing (the committed ./robots.txt), but -test must not be indexed (Disallow: /). With a single tracked file, the only way to get that difference today is to hand-edit the file on the server, which is precisely what blocked checkout.

⚠️ This will recur on the next tagged release

Nothing structural has changed yet. As long as robots.txt is git-tracked and gets per-server edits, the next git checkout tags/<release> can be blocked the same way. We were stuck on 2.5.0 for several releases because of this.

Recommended durable fix (self-service, no Apache/CSE-IT change) — deferred to a later session

makeabilitylabwebsite/rebuildanddeploy.sh already branches per host (DJANGO_ENV=TEST vs PROD) and runs on the host after checkout, where Apache serves the static file. Generate robots.txt there instead of tracking it:

1. git rm --cached robots.txt + add robots.txt to .gitignore → no longer tracked, so checkout can never be blocked by it again.
2. Commit templates, e.g. deploy/robots.prod.txt (allow + Sitemap:) and deploy/robots.test.txt (Disallow: /).
3. In rebuildanddeploy.sh, each host branch copies the matching template to ./robots.txt.

Transition caveat: the first deploy under the new scheme must start from a clean tree. Prod is clean post-merge, but the test server may still carry a local Disallow edit that needs one git stash / git checkout -- robots.txt before the new scheme takes over — worth coordinating with Jason on that single deploy.

Still open

• Implement the durable per-host robots.txt fix above (separate PR).
• CSE IT: restore maintainer access to the build log (un-forbid /logs/buildlog.txt, or mount it to /cse/web/research/makelab/www/ alongside debug.log) — the original blind-spot that made this hard to diagnose.
• Re-run the sitemap crawl validation against production now that it serves 2.9.0, then submit the sitemap in Google Search Console.