diff --git a/.github/README.md b/.github/README.md index 5456bd0..ebe5c59 100644 --- a/.github/README.md +++ b/.github/README.md @@ -13,13 +13,14 @@ GitHub Actions and CI/CD helpers for this repository. | File | Role | |------|------| | [`workflows/ci.yml`](workflows/ci.yml) | Umbrella **CI** — runs on push/PR to `main` and `develop` | -| [`workflows/cd.yml`](workflows/cd.yml) | **Deploy** — after CI succeeds on `develop` (staging); no `workflow_dispatch` trigger | +| [`workflows/cd.yml`](workflows/cd.yml) | **Deploy** — after CI succeeds on push to `develop` (`staging`) or `main` (`production`); inline SSH script parameterized by branch | +| [`workflows/promote-main.yml`](workflows/promote-main.yml) | **Promote to production** — manual `workflow_dispatch`; ff-only `develop` → `main` via `PROMOTE_PAT` so CI and `cd.yml` run on `main` | | [`workflows/release.yml`](workflows/release.yml) | **Release** — manual `workflow_dispatch` only; tags `main` from `pyproject.toml` (`v`) and creates a GitHub Release with Weblate compatibility metadata | | [`workflows/ci-lint.yml`](workflows/ci-lint.yml) | Lint and format (prek) | | [`workflows/ci-test.yml`](workflows/ci-test.yml) | Unit tests and coverage | | [`workflows/ci-package.yml`](workflows/ci-package.yml) | Build and package checks | | [`workflows/ci-dependencies.yml`](workflows/ci-dependencies.yml) | Dependency and license audit | -| [`workflows/ci-weblate-pin.yml`](workflows/ci-weblate-pin.yml) | PyPI vs Docker Weblate pin sync check | +| [`workflows/ci-weblate-pin.yml`](workflows/ci-weblate-pin.yml) | **Weblate version sync** — callable from CI; runs [`scripts/check-weblate-pin-sync.sh`](../scripts/check-weblate-pin-sync.sh) so `pyproject.toml` and `Dockerfile.weblate-plugin` pins match | | [`workflows/weblate-pin-bump.yml`](workflows/weblate-pin-bump.yml) | Scheduled Weblate pin bump (PyPI + Docker + `uv.lock`) | | [`workflows/ci-plugin-smoke.yml`](workflows/ci-plugin-smoke.yml) | Plugin smoke (Docker stack) | | [`workflows/ci-plugin-functional.yml`](workflows/ci-plugin-functional.yml) | Plugin functional tests | @@ -33,7 +34,31 @@ Callable workflows (`ci-*`, `ci-plugin-*`) are triggered only via `workflow_call |------|------| | [`ci/apt-install`](ci/apt-install) | Apt packages for Weblate-related CI jobs | -Deploy uses environment **staging** secrets (`SSH_HOST`, `SSH_USER`, `SSH_PRIVATE_KEY`, `WEBLATE_PORT`, `WEBLATE_URL_PREFIX`) and [`docker/docker-compose.cd.yml`](../docker/docker-compose.cd.yml) on the server at `/opt/cppa-weblate-plugin`. +### Deploy environments and secrets + +[`cd.yml`](workflows/cd.yml) selects the GitHub environment from the CI branch (`workflow_run.head_branch`): + +| Environment | CI branch | When deploy runs | +|-------------|-----------|------------------| +| **staging** | `develop` | After a successful CI run on a push to `develop` | +| **production** | `main` | After a successful CI run on a push to `main` (typically following [`promote-main.yml`](workflows/promote-main.yml)) | + +Both environments use the **same secret names** (configure different values per host): + +| Secret | Purpose | +|--------|---------| +| `SSH_HOST` | Deploy server hostname | +| `SSH_USER` | SSH user | +| `SSH_PRIVATE_KEY` | Private key for deploy | +| `WEBLATE_PORT` | Host port for post-deploy `/healthz/` poll | +| `WEBLATE_URL_PREFIX` | URL prefix for health check (e.g. `/weblate`) | +| `SSH_PORT` | Optional SSH port (default `22`) | + +Server path: `/opt/cppa-weblate-plugin` with [`docker/docker-compose.cd.yml`](../docker/docker-compose.cd.yml). Full procedure: [`docs/deployment-runbook.md`](../docs/deployment-runbook.md). + +### Production promotion (repository secret) + +[`promote-main.yml`](workflows/promote-main.yml) is separate from deploy: it ff-only merges `develop` into `main` and pushes with **`PROMOTE_PAT`** (classic or fine-grained PAT, **Contents: write**). Without a PAT, GitHub does not trigger CI or `cd.yml` on that push. Optional: required reviewers on the **production** environment only. ## Weblate version pinning diff --git a/.github/workflows/cd.yml b/.github/workflows/cd.yml index 636c594..b9b3b7b 100644 --- a/.github/workflows/cd.yml +++ b/.github/workflows/cd.yml @@ -8,10 +8,11 @@ name: CD # - permissions: contents: read (no write access) # - job-level 'if' requires conclusion == 'success' AND event == 'push' # (ignores PRs, forks, and failed/cancelled runs) +# - head_branch limited to main or develop on: workflow_run: workflows: [CI] - branches: [develop] + branches: [main, develop] types: [completed] permissions: @@ -26,26 +27,32 @@ jobs: if: >- github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push' + && (github.event.workflow_run.head_branch == 'main' + || github.event.workflow_run.head_branch == 'develop') runs-on: ubuntu-latest timeout-minutes: 20 - environment: staging + environment: >- + ${{ github.event.workflow_run.head_branch == 'main' && 'production' + || 'staging' }} steps: - name: Deploy via SSH uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2 env: + BRANCH: ${{ github.event.workflow_run.head_branch }} WEBLATE_PORT: ${{ secrets.WEBLATE_PORT }} WEBLATE_URL_PREFIX: ${{ secrets.WEBLATE_URL_PREFIX }} with: host: ${{ secrets.SSH_HOST }} username: ${{ secrets.SSH_USER }} key: ${{ secrets.SSH_PRIVATE_KEY }} - envs: WEBLATE_PORT,WEBLATE_URL_PREFIX + port: ${{ secrets.SSH_PORT || '22' }} + envs: BRANCH,WEBLATE_PORT,WEBLATE_URL_PREFIX script: | set -euo pipefail cd /opt/cppa-weblate-plugin - git fetch origin develop - git checkout develop - git pull origin develop + git fetch origin "$BRANCH" + git checkout "$BRANCH" + git pull origin "$BRANCH" docker compose -f docker/docker-compose.cd.yml --env-file .env build docker compose -f docker/docker-compose.cd.yml --env-file .env up -d WEBLATE_PORT="${WEBLATE_PORT:-8080}" diff --git a/.github/workflows/promote-main.yml b/.github/workflows/promote-main.yml new file mode 100644 index 0000000..5225abf --- /dev/null +++ b/.github/workflows/promote-main.yml @@ -0,0 +1,42 @@ +# SPDX-FileCopyrightText: 2026 William Jin +# +# SPDX-License-Identifier: BSL-1.0 + +# Manual promotion: fast-forward main to match develop (--ff-only). +# Fails if main has diverged (no merge commit; main tip equals develop after success). +# +# Push uses PROMOTE_PAT so CI and CD (workflow_run) still run — GITHUB_TOKEN +# pushes do not trigger follow-on workflows. +# +# Requires repository secret: PROMOTE_PAT (classic or fine-grained PAT with Contents: write). + +name: Promote develop to main + +on: + workflow_dispatch: + +concurrency: + group: promote-production + cancel-in-progress: false + +jobs: + merge: + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + contents: read + steps: + # actions/checkout v6.0.2 + - uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd + with: + ref: main + fetch-depth: 0 + token: ${{ secrets.PROMOTE_PAT }} + persist-credentials: true + + - name: Fast-forward main to develop and push + run: | + set -euo pipefail + git fetch origin develop + git merge --ff-only origin/develop + git push origin main diff --git a/README.md b/README.md index 46e6fca..997d760 100644 --- a/README.md +++ b/README.md @@ -44,9 +44,9 @@ docker compose -f docker/docker-compose.ci.yml up -d Weblate is available at `http://localhost:8080` once the healthcheck passes (admin / `admin`). The CI stack uses ephemeral Postgres on tmpfs — data does not persist across restarts. -### Docker (CD / staging) +### Docker (CD / staging and production) -For persistent deployment with host PostgreSQL and shared Redis, see [docs/deployment-runbook.md](docs/deployment-runbook.md). Quick version: +For persistent deployment with host PostgreSQL and shared Redis, see [docs/deployment-runbook.md](docs/deployment-runbook.md) (staging auto-deploy on `develop`, production via promote + `main` deploy). Quick version: ```bash cp .env.example .env # fill in secrets; see .env.example comments @@ -78,7 +78,7 @@ flowchart TB FMT --> CF FMT --> UTL INST --> EP - EP -->|AppConfig.ready()| RP + EP -->|"AppConfig ready"| RP EP -->|add-or-update| CEL CEL --> EP ``` @@ -142,11 +142,11 @@ Note that adding the app to `INSTALLED_APPS` (by either method) is **necessary b The plugin exposes three HTTP endpoints, all under the `/boost-endpoint/` prefix on the Weblate site: -| Method | Path | Handler | Auth | Response | -|--------|------|---------|------|----------| -| `GET` | `/boost-endpoint/plugin-ping/` | `plugin_ping` | None | `200 ok` (plain text) | -| `GET` | `/boost-endpoint/info/` | `BoostEndpointInfo` | Required | `200` JSON: `module`, `version`, `capabilities` | -| `POST` | `/boost-endpoint/add-or-update/` | `AddOrUpdateView` | Required | `202` JSON: `status`, `task_id`, `detail` | +| Method | Path | Handler | Auth | Rate limit | Response | +|--------|------|---------|------|------------|----------| +| `GET` | `/boost-endpoint/plugin-ping/` | `plugin_ping` | None | None | `200 ok` (plain text) | +| `GET` | `/boost-endpoint/info/` | `BoostEndpointInfo` | Required | Scoped `info` (+ Weblate `UserRateThrottle`) | `200` JSON: `module`, `version`, `capabilities`; `429` with `Retry-After` when throttled | +| `POST` | `/boost-endpoint/add-or-update/` | `AddOrUpdateView` | Required | Scoped `add-or-update` (+ `UserRateThrottle`) | `202` JSON: `status`, `task_id`, `detail`; `429` with `Retry-After` when throttled | When `WEBLATE_URL_PREFIX` is set (e.g. `/weblate`), all paths are prefixed accordingly: `/weblate/boost-endpoint/plugin-ping/`, etc. @@ -202,6 +202,30 @@ The operation is idempotent (guarded by a `_cppa_boost_weblate_urls_registered` The view validates the request with `AddOrUpdateRequestSerializer`, dispatches the Celery task, and returns immediately. A `400` response with an `errors` object is returned if validation fails. +## Rate Limiting + +Protected Boost endpoint views use Django REST Framework throttling merged in [`src/boost_weblate/settings_override.py`](src/boost_weblate/settings_override.py): + +| Scope | Default rate | View | Throttle classes | +|-------|--------------|------|------------------| +| `info` | `60/minute` | `BoostEndpointInfo` | `UserRateThrottle`, `BoostEndpointInfoThrottle` | +| `add-or-update` | `10/hour` | `AddOrUpdateView` | `UserRateThrottle`, `AddOrUpdateThrottle` | + +`BoostEndpointInfoThrottle` and `AddOrUpdateThrottle` subclass DRF `ScopedRateThrottle` and use Weblate’s `@patch_throttle_request` so throttling respects upstream request context. + +**Configuration:** defaults live in `settings_override.py`. Override at deploy time with environment variables (read when the override `exec()` runs): + +| Variable | Scope | Default | +|----------|-------|---------| +| `BOOST_ENDPOINT_THROTTLE_INFO` | `info` | `60/minute` | +| `BOOST_ENDPOINT_THROTTLE_ADD_OR_UPDATE` | `add-or-update` | `10/hour` | + +Rates are merged into `REST_FRAMEWORK["DEFAULT_THROTTLE_RATES"]` via `merge_boost_endpoint_throttle_rates()`. + +**When limited:** clients receive HTTP **`429 Too Many Requests`**. Responses include a **`Retry-After`** header (seconds until retry). DRF may also include wait time in the JSON `errors` detail. `plugin-ping` is not throttled. + +Plugin tests: [`tests/plugin/test_rate_limit.py`](tests/plugin/test_rate_limit.py). + ## Celery Requirement for add-or-update The `POST /boost-endpoint/add-or-update/` endpoint **requires a running Celery worker**. The view enqueues `boost_add_or_update_task` via `.delay()` and returns HTTP 202 immediately — if no worker is consuming the queue, the task sits indefinitely. @@ -257,7 +281,7 @@ The service has no plugin-owned models; it operates entirely through Weblate's D ### CI (`ci.yml`) -Triggered on push and PR to `main` and `develop`. Calls seven reusable sub-workflows: +Triggered on push and PR to `main` and `develop`. Calls eight reusable sub-workflows: | Job | Workflow | What it checks | |-----|----------|----------------| @@ -265,17 +289,36 @@ Triggered on push and PR to `main` and `develop`. Calls seven reusable sub-workf | `test` | [`.github/workflows/ci-test.yml`](.github/workflows/ci-test.yml) | pytest + 90% coverage gate (`--cov-fail-under=90`) | | `package` | [`.github/workflows/ci-package.yml`](.github/workflows/ci-package.yml) | `uv build`, twine, pydistcheck, pyroma, check-wheel-contents, check-manifest | | `dependencies` | [`.github/workflows/ci-dependencies.yml`](.github/workflows/ci-dependencies.yml) | pip-audit, liccheck, dependency review (on PRs) | +| `weblate-pin` | [`.github/workflows/ci-weblate-pin.yml`](.github/workflows/ci-weblate-pin.yml) | PyPI `Weblate[all]==…` in `pyproject.toml` matches Docker `FROM weblate/weblate:…` (`scripts/check-weblate-pin-sync.sh`) | | `plugin-smoke` | [`.github/workflows/ci-plugin-smoke.yml`](.github/workflows/ci-plugin-smoke.yml) | Docker stack → P0 smoke tests (`scripts/plugin-smoke.sh`) | | `plugin-auth` | [`.github/workflows/ci-plugin-auth.yml`](.github/workflows/ci-plugin-auth.yml) | Docker stack → auth tests (`scripts/plugin-auth.sh`) | | `plugin-functional` | [`.github/workflows/ci-plugin-functional.yml`](.github/workflows/ci-plugin-functional.yml) | Docker stack → E2E functional tests (`scripts/plugin-functional.sh`); optional `GH_TEST_REPO_TOKEN` secret for GitHub-backed tests | All `ci-plugin-*` jobs build the CI Docker stack (`docker/docker-compose.ci.yml`), wait for the healthcheck, create an API token, run the corresponding pytest suite under `tests/plugin/`, and tear down. -### CD (`cd.yml`) +[`weblate-pin-bump.yml`](.github/workflows/weblate-pin-bump.yml) runs on a schedule (Monday 09:00 UTC) and opens a PR when a newer PyPI Weblate release has a matching Docker fixed tag. See [`.github/README.md`](.github/README.md#weblate-version-pinning). + +### CD (`cd.yml`) and production promotion (`promote-main.yml`) + +[`cd.yml`](.github/workflows/cd.yml) deploys after a successful **CI** run on a **push** to `develop` or `main`. The GitHub environment and git branch on the server match the CI branch: + +| Path | Trigger | Environment | Server branch | +|------|---------|-------------|---------------| +| **Staging** | Push to `develop` → CI → `cd.yml` | `staging` | `develop` | +| **Production** | [`promote-main.yml`](.github/workflows/promote-main.yml) → CI on `main` → `cd.yml` | `production` | `main` | + +Each deploy SSHes to `/opt/cppa-weblate-plugin`, pulls the branch, rebuilds with `docker/docker-compose.cd.yml`, brings the stack up, and polls `${WEBLATE_URL_PREFIX}/healthz/` on `WEBLATE_PORT` for up to 180 s. On failure, logs the last 40 lines and exits non-zero. Concurrency is locked per branch (`deploy-`). + +**Staging** is fully automatic on `develop` pushes. + +**Production** uses two steps: + +1. **Promote** — run **Actions → Promote develop to main** ([`promote-main.yml`](.github/workflows/promote-main.yml)): fast-forward `main` to `develop` and push with repository secret **`PROMOTE_PAT`** (required so CI and deploy workflows run; `GITHUB_TOKEN` pushes do not trigger them). +2. **Deploy** — when CI on `main` succeeds, `cd.yml` deploys using **production** environment secrets (`SSH_HOST`, `SSH_USER`, `SSH_PRIVATE_KEY`, `WEBLATE_PORT`, `WEBLATE_URL_PREFIX`; optional `SSH_PORT`). -Triggered after CI succeeds on a `develop` push. SSHes into the staging server at `/opt/cppa-weblate-plugin`, pulls the latest code, rebuilds the CD Docker image (`docker/docker-compose.cd.yml`), brings the stack up, and polls `${WEBLATE_URL_PREFIX}/healthz/` on `WEBLATE_PORT` (from `.env`) for up to 180 s. On failure, logs the last 40 lines and exits non-zero. Concurrency is locked per branch so deploys never overlap. +**Release** tagging ([`release.yml`](.github/workflows/release.yml)) is independent of deploy — run manually when you want a GitHub Release on `main`. -Full deployment procedure: [docs/deployment-runbook.md](docs/deployment-runbook.md). +Full deployment and promotion procedure: [docs/deployment-runbook.md](docs/deployment-runbook.md) (staging, production, `PROMOTE_PAT`, rollback, release tagging). ### Running plugin tests locally @@ -299,11 +342,14 @@ Each script builds `docker/docker-compose.ci.yml`, waits for health, runs its py | Topic | File | Description | |-------|------|-------------| | All env vars | [`.env.example`](.env.example) | Annotated template — copy to `.env` on the deploy server | -| Deployment steps | [`docs/deployment-runbook.md`](docs/deployment-runbook.md) | Install, env vars, health checks, troubleshooting | +| Deployment & promotion | [`docs/deployment-runbook.md`](docs/deployment-runbook.md) | Staging/production CD, `PROMOTE_PAT`, environments, health checks, rollback, release tagging | +| Boost endpoint throttles | [`src/boost_weblate/settings_override.py`](src/boost_weblate/settings_override.py) | `BOOST_ENDPOINT_THROTTLE_INFO`, `BOOST_ENDPOINT_THROTTLE_ADD_OR_UPDATE`; merged into `REST_FRAMEWORK` | +| Weblate version pins | [`pyproject.toml`](pyproject.toml), [`docker/Dockerfile.weblate-plugin`](docker/Dockerfile.weblate-plugin) | PyPI and Docker pins kept in sync; CI [`ci-weblate-pin.yml`](.github/workflows/ci-weblate-pin.yml); scheduled bumps via [`weblate-pin-bump.yml`](.github/workflows/weblate-pin-bump.yml) | +| Weblate pin scripts | [`scripts/weblate-version-map.sh`](scripts/weblate-version-map.sh), [`scripts/check-weblate-pin-sync.sh`](scripts/check-weblate-pin-sync.sh) | Calver mapping; CI check via [`ci-weblate-pin.yml`](.github/workflows/ci-weblate-pin.yml) | | API reference | [`docs/boost-endpoint-api.md`](docs/boost-endpoint-api.md) | Full request/response docs for the Boost endpoint | | Route registration | [`docs/plugin-http-routes.md`](docs/plugin-http-routes.md) | How and why routes are registered at startup | | Docker files | [`docker/README.md`](docker/README.md) | Dockerfile and Compose usage for CI and CD | -| CI/CD workflows | [`.github/README.md`](.github/README.md) | Workflow index and secrets reference | +| CI/CD workflows | [`.github/README.md`](.github/README.md) | Workflow index, staging/production secrets, `PROMOTE_PAT` | ## Contributing diff --git a/docs/deployment-runbook.md b/docs/deployment-runbook.md index 0cfd7cc..a95c3fd 100644 --- a/docs/deployment-runbook.md +++ b/docs/deployment-runbook.md @@ -211,15 +211,77 @@ docker compose -f docker/docker-compose.cd.yml --env-file .env \ ## Automated CD Flow -The full pipeline (`cd.yml`) triggers on a successful CI run against `develop`: +Deploy is handled by [`cd.yml`](../.github/workflows/cd.yml) after a successful **CI** run on a **push** to `develop` or `main`. The GitHub environment (`staging` or `production`) and git branch on the server follow the CI branch. -1. SSH to the deploy server -2. `git pull origin develop` -3. `docker compose … build && up -d` +| Branch | Trigger | GitHub environment | Server branch | +|--------|---------|-------------------|---------------| +| `develop` | Push to `develop` → CI → `cd.yml` | `staging` | `develop` | +| `main` | Promote (below) → CI on `main` → `cd.yml` | `production` | `main` | + +Each deploy job: + +1. SSH to the deploy server (`/opt/cppa-weblate-plugin`) +2. `git fetch` / `checkout` / `pull` the CI branch +3. `docker compose -f docker/docker-compose.cd.yml --env-file .env build && up -d` 4. Poll `${WEBLATE_URL_PREFIX}/healthz/` on `WEBLATE_PORT` for up to 180 s 5. On failure: dump the last 40 lines of container logs and exit non-zero -Concurrency is locked per branch (`cancel-in-progress: false`) so deploys never overlap. +Concurrency is locked per branch (`cancel-in-progress: false`) so staging and production deploys do not overlap on the same branch group. + +### Staging (`develop`) + +Merge or push to `develop`. When CI succeeds, `cd.yml` deploys using **staging** environment secrets. + +### Production (`main`) + +1. Validate on staging (`develop` CI + deploy). +2. Ensure `main` can fast-forward to `develop` (`main` is an ancestor of `develop`, or equal). +3. Run **Actions → Promote develop to main** ([`promote-main.yml`](../.github/workflows/promote-main.yml)). +4. The workflow ff-only merges `origin/develop` into `main` and pushes with **`PROMOTE_PAT`** (repository secret). +5. That push runs CI on `main`; when CI succeeds, `cd.yml` deploys using **production** environment secrets. + +#### Why `PROMOTE_PAT` is required + +`promote-main.yml` is started with `workflow_dispatch`, but the push to `main` must use a **PAT**, not the default `GITHUB_TOKEN`. GitHub does not run `push`-triggered workflows (including CI and `cd.yml`’s `workflow_run`) for commits pushed with `GITHUB_TOKEN`. + +Configure a classic or fine-grained PAT with **Contents: write** on this repository and store it as the **`PROMOTE_PAT`** repository secret. + +#### Fast-forward failure + +If `main` has diverged from `develop`, `git merge --ff-only` fails. Resolve locally (rebase or reset `main` to match your release policy), then re-run the promote workflow. + +### GitHub environments and secrets + +| Environment | Used when | Secrets (same names per environment) | +|-------------|-----------|-------------------------------------| +| `staging` | CI on `develop` | `SSH_HOST`, `SSH_USER`, `SSH_PRIVATE_KEY`, `WEBLATE_PORT`, `WEBLATE_URL_PREFIX`; optional `SSH_PORT` | +| `production` | CI on `main` | Same names; production host values | + +Optional: enable required reviewers on the `production` environment. + +### Rollback (production or staging) + +On the deploy server: + +```bash +cd /opt/cppa-weblate-plugin +git fetch origin +git checkout +docker compose -f docker/docker-compose.cd.yml --env-file .env build +docker compose -f docker/docker-compose.cd.yml --env-file .env up -d +# Re-run health poll from "External health poll" above +``` + +Reverting the server does not automatically move `main` or `develop` on GitHub; fix branch tips separately if needed. + +### CD failure modes + +| Failure | Likely cause | +|---------|----------------| +| Deploy skipped after promote | `PROMOTE_PAT` missing or push used `GITHUB_TOKEN`; CI on `main` never ran | +| FF-only merge failed | `main` diverged from `develop` | +| Health check timeout | Weblate boot/migrations, Postgres, Redis, or URL prefix mismatch | +| Wrong environment deployed | CI ran on unexpected branch; check workflow run `head_branch` | ## Release tagging