Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 28 additions & 3 deletions .github/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,14 @@ GitHub Actions and CI/CD helpers for this repository.
| File | Role |
|------|------|
| [`workflows/ci.yml`](workflows/ci.yml) | Umbrella **CI** — runs on push/PR to `main` and `develop` |
| [`workflows/cd.yml`](workflows/cd.yml) | **Deploy** — after CI succeeds on `develop` (staging); no `workflow_dispatch` trigger |
| [`workflows/cd.yml`](workflows/cd.yml) | **Deploy** — after CI succeeds on push to `develop` (`staging`) or `main` (`production`); inline SSH script parameterized by branch |
| [`workflows/promote-main.yml`](workflows/promote-main.yml) | **Promote to production** — manual `workflow_dispatch`; ff-only `develop` → `main` via `PROMOTE_PAT` so CI and `cd.yml` run on `main` |
| [`workflows/release.yml`](workflows/release.yml) | **Release** — manual `workflow_dispatch` only; tags `main` from `pyproject.toml` (`v<version>`) and creates a GitHub Release with Weblate compatibility metadata |
| [`workflows/ci-lint.yml`](workflows/ci-lint.yml) | Lint and format (prek) |
| [`workflows/ci-test.yml`](workflows/ci-test.yml) | Unit tests and coverage |
| [`workflows/ci-package.yml`](workflows/ci-package.yml) | Build and package checks |
| [`workflows/ci-dependencies.yml`](workflows/ci-dependencies.yml) | Dependency and license audit |
| [`workflows/ci-weblate-pin.yml`](workflows/ci-weblate-pin.yml) | PyPI vs Docker Weblate pin sync check |
| [`workflows/ci-weblate-pin.yml`](workflows/ci-weblate-pin.yml) | **Weblate version sync** — callable from CI; runs [`scripts/check-weblate-pin-sync.sh`](../scripts/check-weblate-pin-sync.sh) so `pyproject.toml` and `Dockerfile.weblate-plugin` pins match |
| [`workflows/weblate-pin-bump.yml`](workflows/weblate-pin-bump.yml) | Scheduled Weblate pin bump (PyPI + Docker + `uv.lock`) |
| [`workflows/ci-plugin-smoke.yml`](workflows/ci-plugin-smoke.yml) | Plugin smoke (Docker stack) |
| [`workflows/ci-plugin-functional.yml`](workflows/ci-plugin-functional.yml) | Plugin functional tests |
Expand All @@ -33,7 +34,31 @@ Callable workflows (`ci-*`, `ci-plugin-*`) are triggered only via `workflow_call
|------|------|
| [`ci/apt-install`](ci/apt-install) | Apt packages for Weblate-related CI jobs |

Deploy uses environment **staging** secrets (`SSH_HOST`, `SSH_USER`, `SSH_PRIVATE_KEY`, `WEBLATE_PORT`, `WEBLATE_URL_PREFIX`) and [`docker/docker-compose.cd.yml`](../docker/docker-compose.cd.yml) on the server at `/opt/cppa-weblate-plugin`.
### Deploy environments and secrets

[`cd.yml`](workflows/cd.yml) selects the GitHub environment from the CI branch (`workflow_run.head_branch`):

| Environment | CI branch | When deploy runs |
|-------------|-----------|------------------|
| **staging** | `develop` | After a successful CI run on a push to `develop` |
| **production** | `main` | After a successful CI run on a push to `main` (typically following [`promote-main.yml`](workflows/promote-main.yml)) |

Both environments use the **same secret names** (configure different values per host):

| Secret | Purpose |
|--------|---------|
| `SSH_HOST` | Deploy server hostname |
| `SSH_USER` | SSH user |
| `SSH_PRIVATE_KEY` | Private key for deploy |
| `WEBLATE_PORT` | Host port for post-deploy `/healthz/` poll |
| `WEBLATE_URL_PREFIX` | URL prefix for health check (e.g. `/weblate`) |
| `SSH_PORT` | Optional SSH port (default `22`) |

Server path: `/opt/cppa-weblate-plugin` with [`docker/docker-compose.cd.yml`](../docker/docker-compose.cd.yml). Full procedure: [`docs/deployment-runbook.md`](../docs/deployment-runbook.md).

### Production promotion (repository secret)

[`promote-main.yml`](workflows/promote-main.yml) is separate from deploy: it ff-only merges `develop` into `main` and pushes with **`PROMOTE_PAT`** (classic or fine-grained PAT, **Contents: write**). Without a PAT, GitHub does not trigger CI or `cd.yml` on that push. Optional: required reviewers on the **production** environment only.

## Weblate version pinning

Expand Down
19 changes: 13 additions & 6 deletions .github/workflows/cd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,11 @@ name: CD
# - permissions: contents: read (no write access)
# - job-level 'if' requires conclusion == 'success' AND event == 'push'
# (ignores PRs, forks, and failed/cancelled runs)
# - head_branch limited to main or develop
on:
workflow_run:
workflows: [CI]
branches: [develop]
branches: [main, develop]
types: [completed]

permissions:
Expand All @@ -26,26 +27,32 @@ jobs:
if: >-
github.event.workflow_run.conclusion == 'success'
&& github.event.workflow_run.event == 'push'
&& (github.event.workflow_run.head_branch == 'main'
|| github.event.workflow_run.head_branch == 'develop')
runs-on: ubuntu-latest
timeout-minutes: 20
environment: staging
environment: >-
${{ github.event.workflow_run.head_branch == 'main' && 'production'
|| 'staging' }}
steps:
- name: Deploy via SSH
uses: appleboy/ssh-action@0ff4204d59e8e51228ff73bce53f80d53301dee2
env:
BRANCH: ${{ github.event.workflow_run.head_branch }}
WEBLATE_PORT: ${{ secrets.WEBLATE_PORT }}
WEBLATE_URL_PREFIX: ${{ secrets.WEBLATE_URL_PREFIX }}
with:
host: ${{ secrets.SSH_HOST }}
username: ${{ secrets.SSH_USER }}
key: ${{ secrets.SSH_PRIVATE_KEY }}
envs: WEBLATE_PORT,WEBLATE_URL_PREFIX
port: ${{ secrets.SSH_PORT || '22' }}
envs: BRANCH,WEBLATE_PORT,WEBLATE_URL_PREFIX
script: |
set -euo pipefail
cd /opt/cppa-weblate-plugin
git fetch origin develop
git checkout develop
git pull origin develop
git fetch origin "$BRANCH"
git checkout "$BRANCH"
git pull origin "$BRANCH"
docker compose -f docker/docker-compose.cd.yml --env-file .env build
docker compose -f docker/docker-compose.cd.yml --env-file .env up -d
WEBLATE_PORT="${WEBLATE_PORT:-8080}"
Expand Down
42 changes: 42 additions & 0 deletions .github/workflows/promote-main.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
# SPDX-FileCopyrightText: 2026 William Jin <AuraMindNest@outlook.com>
#
# SPDX-License-Identifier: BSL-1.0

# Manual promotion: fast-forward main to match develop (--ff-only).
# Fails if main has diverged (no merge commit; main tip equals develop after success).
#
# Push uses PROMOTE_PAT so CI and CD (workflow_run) still run — GITHUB_TOKEN
# pushes do not trigger follow-on workflows.
#
# Requires repository secret: PROMOTE_PAT (classic or fine-grained PAT with Contents: write).

name: Promote develop to main

on:
workflow_dispatch:

concurrency:
group: promote-production
cancel-in-progress: false

jobs:
merge:
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
steps:
# actions/checkout v6.0.2
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd
with:
ref: main
fetch-depth: 0
token: ${{ secrets.PROMOTE_PAT }}
persist-credentials: true

- name: Fast-forward main to develop and push
run: |
set -euo pipefail
git fetch origin develop
git merge --ff-only origin/develop
git push origin main
74 changes: 60 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,9 +44,9 @@ docker compose -f docker/docker-compose.ci.yml up -d

Weblate is available at `http://localhost:8080` once the healthcheck passes (admin / `admin`). The CI stack uses ephemeral Postgres on tmpfs — data does not persist across restarts.

### Docker (CD / staging)
### Docker (CD / staging and production)

For persistent deployment with host PostgreSQL and shared Redis, see [docs/deployment-runbook.md](docs/deployment-runbook.md). Quick version:
For persistent deployment with host PostgreSQL and shared Redis, see [docs/deployment-runbook.md](docs/deployment-runbook.md) (staging auto-deploy on `develop`, production via promote + `main` deploy). Quick version:

```bash
cp .env.example .env # fill in secrets; see .env.example comments
Expand Down Expand Up @@ -78,7 +78,7 @@ flowchart TB
FMT --> CF
FMT --> UTL
INST --> EP
EP -->|AppConfig.ready()| RP
EP -->|"AppConfig ready"| RP
EP -->|add-or-update| CEL
CEL --> EP
```
Expand Down Expand Up @@ -142,11 +142,11 @@ Note that adding the app to `INSTALLED_APPS` (by either method) is **necessary b

The plugin exposes three HTTP endpoints, all under the `/boost-endpoint/` prefix on the Weblate site:

| Method | Path | Handler | Auth | Response |
|--------|------|---------|------|----------|
| `GET` | `/boost-endpoint/plugin-ping/` | `plugin_ping` | None | `200 ok` (plain text) |
| `GET` | `/boost-endpoint/info/` | `BoostEndpointInfo` | Required | `200` JSON: `module`, `version`, `capabilities` |
| `POST` | `/boost-endpoint/add-or-update/` | `AddOrUpdateView` | Required | `202` JSON: `status`, `task_id`, `detail` |
| Method | Path | Handler | Auth | Rate limit | Response |
|--------|------|---------|------|------------|----------|
| `GET` | `/boost-endpoint/plugin-ping/` | `plugin_ping` | None | None | `200 ok` (plain text) |
| `GET` | `/boost-endpoint/info/` | `BoostEndpointInfo` | Required | Scoped `info` (+ Weblate `UserRateThrottle`) | `200` JSON: `module`, `version`, `capabilities`; `429` with `Retry-After` when throttled |
| `POST` | `/boost-endpoint/add-or-update/` | `AddOrUpdateView` | Required | Scoped `add-or-update` (+ `UserRateThrottle`) | `202` JSON: `status`, `task_id`, `detail`; `429` with `Retry-After` when throttled |

When `WEBLATE_URL_PREFIX` is set (e.g. `/weblate`), all paths are prefixed accordingly: `/weblate/boost-endpoint/plugin-ping/`, etc.

Expand Down Expand Up @@ -202,6 +202,30 @@ The operation is idempotent (guarded by a `_cppa_boost_weblate_urls_registered`

The view validates the request with `AddOrUpdateRequestSerializer`, dispatches the Celery task, and returns immediately. A `400` response with an `errors` object is returned if validation fails.

## Rate Limiting

Protected Boost endpoint views use Django REST Framework throttling merged in [`src/boost_weblate/settings_override.py`](src/boost_weblate/settings_override.py):

| Scope | Default rate | View | Throttle classes |
|-------|--------------|------|------------------|
| `info` | `60/minute` | `BoostEndpointInfo` | `UserRateThrottle`, `BoostEndpointInfoThrottle` |
| `add-or-update` | `10/hour` | `AddOrUpdateView` | `UserRateThrottle`, `AddOrUpdateThrottle` |

`BoostEndpointInfoThrottle` and `AddOrUpdateThrottle` subclass DRF `ScopedRateThrottle` and use Weblate’s `@patch_throttle_request` so throttling respects upstream request context.

**Configuration:** defaults live in `settings_override.py`. Override at deploy time with environment variables (read when the override `exec()` runs):

| Variable | Scope | Default |
|----------|-------|---------|
| `BOOST_ENDPOINT_THROTTLE_INFO` | `info` | `60/minute` |
| `BOOST_ENDPOINT_THROTTLE_ADD_OR_UPDATE` | `add-or-update` | `10/hour` |

Rates are merged into `REST_FRAMEWORK["DEFAULT_THROTTLE_RATES"]` via `merge_boost_endpoint_throttle_rates()`.

**When limited:** clients receive HTTP **`429 Too Many Requests`**. Responses include a **`Retry-After`** header (seconds until retry). DRF may also include wait time in the JSON `errors` detail. `plugin-ping` is not throttled.

Plugin tests: [`tests/plugin/test_rate_limit.py`](tests/plugin/test_rate_limit.py).

## Celery Requirement for add-or-update

The `POST /boost-endpoint/add-or-update/` endpoint **requires a running Celery worker**. The view enqueues `boost_add_or_update_task` via `.delay()` and returns HTTP 202 immediately — if no worker is consuming the queue, the task sits indefinitely.
Expand Down Expand Up @@ -257,25 +281,44 @@ The service has no plugin-owned models; it operates entirely through Weblate's D

### CI (`ci.yml`)

Triggered on push and PR to `main` and `develop`. Calls seven reusable sub-workflows:
Triggered on push and PR to `main` and `develop`. Calls eight reusable sub-workflows:

| Job | Workflow | What it checks |
|-----|----------|----------------|
| `lint` | [`.github/workflows/ci-lint.yml`](.github/workflows/ci-lint.yml) | prek (Ruff, YAML/TOML, REUSE, actionlint, pytest) |
| `test` | [`.github/workflows/ci-test.yml`](.github/workflows/ci-test.yml) | pytest + 90% coverage gate (`--cov-fail-under=90`) |
| `package` | [`.github/workflows/ci-package.yml`](.github/workflows/ci-package.yml) | `uv build`, twine, pydistcheck, pyroma, check-wheel-contents, check-manifest |
| `dependencies` | [`.github/workflows/ci-dependencies.yml`](.github/workflows/ci-dependencies.yml) | pip-audit, liccheck, dependency review (on PRs) |
| `weblate-pin` | [`.github/workflows/ci-weblate-pin.yml`](.github/workflows/ci-weblate-pin.yml) | PyPI `Weblate[all]==…` in `pyproject.toml` matches Docker `FROM weblate/weblate:…` (`scripts/check-weblate-pin-sync.sh`) |
| `plugin-smoke` | [`.github/workflows/ci-plugin-smoke.yml`](.github/workflows/ci-plugin-smoke.yml) | Docker stack → P0 smoke tests (`scripts/plugin-smoke.sh`) |
| `plugin-auth` | [`.github/workflows/ci-plugin-auth.yml`](.github/workflows/ci-plugin-auth.yml) | Docker stack → auth tests (`scripts/plugin-auth.sh`) |
| `plugin-functional` | [`.github/workflows/ci-plugin-functional.yml`](.github/workflows/ci-plugin-functional.yml) | Docker stack → E2E functional tests (`scripts/plugin-functional.sh`); optional `GH_TEST_REPO_TOKEN` secret for GitHub-backed tests |

All `ci-plugin-*` jobs build the CI Docker stack (`docker/docker-compose.ci.yml`), wait for the healthcheck, create an API token, run the corresponding pytest suite under `tests/plugin/`, and tear down.

### CD (`cd.yml`)
[`weblate-pin-bump.yml`](.github/workflows/weblate-pin-bump.yml) runs on a schedule (Monday 09:00 UTC) and opens a PR when a newer PyPI Weblate release has a matching Docker fixed tag. See [`.github/README.md`](.github/README.md#weblate-version-pinning).

### CD (`cd.yml`) and production promotion (`promote-main.yml`)

[`cd.yml`](.github/workflows/cd.yml) deploys after a successful **CI** run on a **push** to `develop` or `main`. The GitHub environment and git branch on the server match the CI branch:

| Path | Trigger | Environment | Server branch |
|------|---------|-------------|---------------|
| **Staging** | Push to `develop` → CI → `cd.yml` | `staging` | `develop` |
| **Production** | [`promote-main.yml`](.github/workflows/promote-main.yml) → CI on `main` → `cd.yml` | `production` | `main` |

Each deploy SSHes to `/opt/cppa-weblate-plugin`, pulls the branch, rebuilds with `docker/docker-compose.cd.yml`, brings the stack up, and polls `${WEBLATE_URL_PREFIX}/healthz/` on `WEBLATE_PORT` for up to 180 s. On failure, logs the last 40 lines and exits non-zero. Concurrency is locked per branch (`deploy-<branch>`).

**Staging** is fully automatic on `develop` pushes.

**Production** uses two steps:

1. **Promote** — run **Actions → Promote develop to main** ([`promote-main.yml`](.github/workflows/promote-main.yml)): fast-forward `main` to `develop` and push with repository secret **`PROMOTE_PAT`** (required so CI and deploy workflows run; `GITHUB_TOKEN` pushes do not trigger them).
2. **Deploy** — when CI on `main` succeeds, `cd.yml` deploys using **production** environment secrets (`SSH_HOST`, `SSH_USER`, `SSH_PRIVATE_KEY`, `WEBLATE_PORT`, `WEBLATE_URL_PREFIX`; optional `SSH_PORT`).

Triggered after CI succeeds on a `develop` push. SSHes into the staging server at `/opt/cppa-weblate-plugin`, pulls the latest code, rebuilds the CD Docker image (`docker/docker-compose.cd.yml`), brings the stack up, and polls `${WEBLATE_URL_PREFIX}/healthz/` on `WEBLATE_PORT` (from `.env`) for up to 180 s. On failure, logs the last 40 lines and exits non-zero. Concurrency is locked per branch so deploys never overlap.
**Release** tagging ([`release.yml`](.github/workflows/release.yml)) is independent of deploy — run manually when you want a GitHub Release on `main`.

Full deployment procedure: [docs/deployment-runbook.md](docs/deployment-runbook.md).
Full deployment and promotion procedure: [docs/deployment-runbook.md](docs/deployment-runbook.md) (staging, production, `PROMOTE_PAT`, rollback, release tagging).

### Running plugin tests locally

Expand All @@ -299,11 +342,14 @@ Each script builds `docker/docker-compose.ci.yml`, waits for health, runs its py
| Topic | File | Description |
|-------|------|-------------|
| All env vars | [`.env.example`](.env.example) | Annotated template — copy to `.env` on the deploy server |
| Deployment steps | [`docs/deployment-runbook.md`](docs/deployment-runbook.md) | Install, env vars, health checks, troubleshooting |
| Deployment & promotion | [`docs/deployment-runbook.md`](docs/deployment-runbook.md) | Staging/production CD, `PROMOTE_PAT`, environments, health checks, rollback, release tagging |
| Boost endpoint throttles | [`src/boost_weblate/settings_override.py`](src/boost_weblate/settings_override.py) | `BOOST_ENDPOINT_THROTTLE_INFO`, `BOOST_ENDPOINT_THROTTLE_ADD_OR_UPDATE`; merged into `REST_FRAMEWORK` |
| Weblate version pins | [`pyproject.toml`](pyproject.toml), [`docker/Dockerfile.weblate-plugin`](docker/Dockerfile.weblate-plugin) | PyPI and Docker pins kept in sync; CI [`ci-weblate-pin.yml`](.github/workflows/ci-weblate-pin.yml); scheduled bumps via [`weblate-pin-bump.yml`](.github/workflows/weblate-pin-bump.yml) |
| Weblate pin scripts | [`scripts/weblate-version-map.sh`](scripts/weblate-version-map.sh), [`scripts/check-weblate-pin-sync.sh`](scripts/check-weblate-pin-sync.sh) | Calver mapping; CI check via [`ci-weblate-pin.yml`](.github/workflows/ci-weblate-pin.yml) |
| API reference | [`docs/boost-endpoint-api.md`](docs/boost-endpoint-api.md) | Full request/response docs for the Boost endpoint |
| Route registration | [`docs/plugin-http-routes.md`](docs/plugin-http-routes.md) | How and why routes are registered at startup |
| Docker files | [`docker/README.md`](docker/README.md) | Dockerfile and Compose usage for CI and CD |
| CI/CD workflows | [`.github/README.md`](.github/README.md) | Workflow index and secrets reference |
| CI/CD workflows | [`.github/README.md`](.github/README.md) | Workflow index, staging/production secrets, `PROMOTE_PAT` |

## Contributing

Expand Down
Loading
Loading