Skip to content

Commit e1be8a0

Browse files
nuclearcatnuclearcat
authored
feat(deploy): migrate compose-next to single-env deployment with profiles and prebuilt images (#1759)
Refactor the deployment flow so docker-compose-next uses one shared environment file and supports cleaner production/staging profiles without local multi-file env wiring. - Replace fragmented env-file usage in docker-compose-next with a unified .env template and sensible defaults for IMAGE_* registry/tag values. - Add local DB profile (`local-db`) and command-enabled profile (`with_commands`) in docker-compose-next, including backend service command/entrypoint definitions. - Add ingester and pending_aggregations worker services behind `with_commands`, with proper SKIP_CRONJOBS configuration and spool/metrics wiring. - Move dashboard_db to profile-gated startup with explicit healthcheck and env-driven credentials for staging/local DB scenarios. - Update backend entrypoint to read `DB_PASSWORD` (supports Docker secret file flow via existing `file_env`) and remove inline `DB_DEFAULT` JSON env generation. - Update `.env.example` into a single, documented template for next-compose usage with clearer variable grouping and placeholders. - Add `DEPLOYMENT.md` documenting development, production, and staging compose flows, plus migration notes from legacy `DB_DEFAULT_*` variables to `DB_*`. Signed-off-by: Denys Fedoryshchenko <denys.f@collabora.com>
1 parent e232e34 commit e1be8a0

8 files changed

Lines changed: 384 additions & 36 deletions

File tree

‎.env.backend.example‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -10,12 +10,12 @@ DEBUG_DB_VARS=${DEBUG_DB_VARS:-False}
1010
ENABLE_LOGGING=${ENABLE_LOGGING:-False} # Logs all prints
1111
DEBUG_SQL_QUERY=${DEBUG_SQL_QUERY:-False} # Logs the SQL queries made
1212

13-
DB_NAME=${DB_DEFAULT_NAME:-dashboard}
14-
DB_USER=${DB_DEFAULT_USER:-admin}
13+
DB_NAME=${DB_NAME:-dashboard}
14+
DB_USER=${DB_USER:-admin}
1515
DB_PASSWORD=db_password
1616
DB_HOST=dashboard_db # Docker can't connect to the ssh tunnel host directly.
17-
DB_PORT=${DB_DEFAULT_PORT:-5432}
18-
DB_ENGINE=${DB_DEFAULT_ENGINE:-django.db.backends.postgresql}
17+
DB_PORT=${DB_PORT:-5432}
18+
DB_ENGINE=${DB_ENGINE:-django.db.backends.postgresql}
1919
DB_OPTIONS_CONNECT_TIMEOUT=${DB_OPTIONS_CONNECT_TIMEOUT:-16}
2020

2121
# Check docs/monitoring.md docs for more context

‎.env.example‎

Lines changed: 84 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,85 @@
1-
# Top-level variables that should be used just for the docker-compose setup.
2-
INGESTER_METRICS_PORT=8002
3-
INGESTER_TREE_NAMES_FILE_DIR=../data
1+
# =============================================================================
2+
# KernelCI Dashboard — Single .env for docker-compose-next.yml
3+
# =============================================================================
4+
# Copy this file to .env and edit the values marked CHANGE_ME.
5+
# See DEPLOYMENT.md for full deployment instructions.
6+
7+
# -----------------------------------------------------------------------------
8+
# Image Configuration
9+
# -----------------------------------------------------------------------------
10+
# Pre-built images are pulled from this registry. Override for custom builds.
11+
IMAGE_REGISTRY=ghcr.io
12+
IMAGE_OWNER=kernelci
13+
IMAGE_REPOSITORY=dashboard
14+
IMAGE_TAG=latest
15+
16+
# -----------------------------------------------------------------------------
17+
# Database
18+
# -----------------------------------------------------------------------------
19+
# For production: point to your external PostgreSQL instance.
20+
# For staging (--profile=local-db): keep DB_HOST=dashboard_db.
21+
DB_HOST=dashboard_db
22+
DB_PORT=5432
23+
DB_NAME=dashboard
24+
DB_USER=admin
25+
DB_PASSWORD=CHANGE_ME
26+
DB_ENGINE=django.db.backends.postgresql
27+
DB_OPTIONS_CONNECT_TIMEOUT=16
28+
29+
# Optional: separate application database user/name for setup-dashboard-db.sh.
30+
# Defaults to DB_USER / DB_NAME if not set.
31+
# APP_DB_USER=dashboard
32+
# APP_DB=dashboard
33+
34+
# -----------------------------------------------------------------------------
35+
# Django
36+
# -----------------------------------------------------------------------------
37+
DJANGO_SECRET_KEY=CHANGE_ME
38+
ALLOWED_HOSTS=["backend", "localhost"]
39+
CORS_ALLOWED_ORIGINS=[]
40+
DEBUG=False
41+
SKIP_CRONJOBS=False
42+
43+
# -----------------------------------------------------------------------------
44+
# Debug Flags (uncomment to enable)
45+
# -----------------------------------------------------------------------------
46+
# DEBUG_DB_VARS=True
47+
# ENABLE_LOGGING=True
48+
# DEBUG_SQL_QUERY=True
49+
50+
# -----------------------------------------------------------------------------
51+
# Redis
52+
# -----------------------------------------------------------------------------
53+
# Always "redis" when running inside Docker Compose.
54+
REDIS_HOST=redis
55+
56+
# -----------------------------------------------------------------------------
57+
# Monitoring (optional)
58+
# -----------------------------------------------------------------------------
59+
# See docs/monitoring.md for details.
60+
PROMETHEUS_METRICS_ENABLED=False
61+
# PROMETHEUS_METRICS_PORT=8001
62+
# PROMETHEUS_MULTIPROC_DIR=/tmp/metrics
63+
64+
# -----------------------------------------------------------------------------
65+
# Email / Notifications (optional)
66+
# -----------------------------------------------------------------------------
67+
# See docs/notifications.md for details.
68+
# EMAIL_HOST=smtp.gmail.com
69+
# EMAIL_PORT=587
70+
# EMAIL_USE_TLS=True
71+
# EMAIL_HOST_USER=bot@kernelci.org
72+
# EMAIL_HOST_PASSWORD=
73+
# DISCORD_WEBHOOK_URL=
74+
75+
# -----------------------------------------------------------------------------
76+
# Backend Volume
77+
# -----------------------------------------------------------------------------
78+
BACKEND_VOLUME_DIR=/volume_data
79+
80+
# -----------------------------------------------------------------------------
81+
# Ingester (only needed with --profile=with_commands)
82+
# -----------------------------------------------------------------------------
83+
# Host path where the ingester monitors for submission files.
484
INGESTER_SPOOL_DIR=../spool
85+
# INGESTER_METRICS_PORT=8002

‎DEPLOYMENT.md‎

Lines changed: 206 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,206 @@
1+
# KernelCI Dashboard — Deployment Guide
2+
3+
This guide covers three deployment scenarios: **development**, **production**, and **staging**.
4+
5+
## Quick Reference
6+
7+
| Scenario | Compose File | Database | Profiles |
8+
|----------|-------------|----------|----------|
9+
| Development | `docker-compose.yml` | Local (always on) | `with_commands` for ingester |
10+
| Production | `docker-compose-next.yml` | External PostgreSQL | none (or `with_commands`) |
11+
| Staging | `docker-compose-next.yml` | Local via profile | `local-db` (+ `with_commands`) |
12+
13+
---
14+
15+
## 1. Development (`docker-compose.yml`)
16+
17+
The development setup builds images locally and uses per-service `.env` files.
18+
19+
### Setup
20+
21+
```bash
22+
# Copy all example env files
23+
cp .env.example .env
24+
cp .env.backend.example .env.backend
25+
cp .env.db.example .env.db
26+
cp .env.proxy.example .env.proxy
27+
cp .env.ingester.example .env.ingester
28+
cp .env.pending_aggregations.example .env.pending_aggregations
29+
30+
# Start all core services (builds images from source)
31+
docker compose up --build -d
32+
33+
# Include ingester and aggregation processor
34+
docker compose --profile=with_commands up --build -d
35+
```
36+
37+
### Rebuilding after code changes
38+
39+
```bash
40+
# Rebuild and restart just the backend
41+
docker compose up --build -d backend
42+
43+
# Rebuild everything
44+
docker compose up --build -d
45+
```
46+
47+
### Frontend development
48+
49+
For active frontend work, run the Vite dev server directly:
50+
51+
```bash
52+
cd dashboard
53+
pnpm install
54+
# Copy the example env file and verify VITE_API_BASE_URL
55+
cp .env.example .env
56+
pnpm dev
57+
```
58+
59+
The frontend connects to the backend API via the `VITE_API_BASE_URL` defined in `dashboard/.env` (defaults to `http://localhost:8000`).
60+
61+
---
62+
63+
## 2. Production (`docker-compose-next.yml`, external PostgreSQL)
64+
65+
Uses pre-built images from GHCR and connects to an external PostgreSQL instance.
66+
67+
### Setup
68+
69+
```bash
70+
# 1. Create .env from the template
71+
cp .env.example .env
72+
73+
# 2. Edit .env — at minimum, set these:
74+
# DB_HOST → your PostgreSQL host
75+
# DB_PORT → your PostgreSQL port (default: 5432)
76+
# DB_PASSWORD → your PostgreSQL password
77+
# DJANGO_SECRET_KEY → a strong random string
78+
# ALLOWED_HOSTS → e.g. ["backend", "your-domain.com"]
79+
# CORS_ALLOWED_ORIGINS → e.g. ["https://your-domain.com"]
80+
81+
# 3. Start services
82+
docker compose -f docker-compose-next.yml up -d
83+
84+
# 4. Verify
85+
curl http://localhost/api/
86+
87+
# 5. Run database migrations (first deploy or after updates)
88+
docker compose -f docker-compose-next.yml run --rm backend \
89+
sh -c "chmod +x ./migrate-app-db.sh && ./migrate-app-db.sh"
90+
```
91+
92+
### With ingester and aggregation processor
93+
94+
```bash
95+
# Set INGESTER_SPOOL_DIR in .env to the host path where submissions arrive
96+
docker compose -f docker-compose-next.yml --profile=with_commands up -d
97+
```
98+
99+
### Updating to a new version
100+
101+
```bash
102+
# Pull latest images and restart
103+
docker compose -f docker-compose-next.yml pull
104+
docker compose -f docker-compose-next.yml up -d
105+
106+
# Run migrations if needed
107+
docker compose -f docker-compose-next.yml run --rm backend \
108+
sh -c "chmod +x ./migrate-app-db.sh && ./migrate-app-db.sh"
109+
```
110+
111+
---
112+
113+
## 3. Staging (`docker-compose-next.yml`, local PostgreSQL)
114+
115+
Uses pre-built images with a local PostgreSQL container via the `local-db` profile.
116+
117+
### Setup
118+
119+
```bash
120+
# 1. Create .env from the template
121+
cp .env.example .env
122+
123+
# 2. Edit .env — at minimum, set these:
124+
# DB_PASSWORD → choose a password for the local postgres
125+
# DJANGO_SECRET_KEY → a random string (can be less strict for staging)
126+
# Keep DB_HOST=dashboard_db (the default)
127+
128+
# 3. Start the database first (wait for it to be ready)
129+
docker compose -f docker-compose-next.yml --profile=local-db up -d dashboard_db
130+
docker compose -f docker-compose-next.yml exec dashboard_db pg_isready -U admin
131+
132+
# 4. Start remaining services
133+
docker compose -f docker-compose-next.yml --profile=local-db up -d
134+
135+
# 5. Verify
136+
curl http://localhost:8000/api/
137+
curl http://localhost/
138+
```
139+
140+
### With all optional services
141+
142+
```bash
143+
docker compose -f docker-compose-next.yml --profile=local-db --profile=with_commands up -d
144+
```
145+
146+
### Tear down (including database volume)
147+
148+
```bash
149+
docker compose -f docker-compose-next.yml --profile=local-db down -v
150+
```
151+
152+
---
153+
154+
## Profile Reference
155+
156+
| Command | Services |
157+
|---------|----------|
158+
| `docker compose -f docker-compose-next.yml up -d` | redis, backend, dashboard, proxy |
159+
| `... --profile=local-db up -d` | + dashboard_db |
160+
| `... --profile=with_commands up -d` | + ingester, pending_aggregations_processor |
161+
| `... --profile=local-db --profile=with_commands up -d` | All services |
162+
163+
---
164+
165+
## Docker Secrets Support
166+
167+
The backend entrypoint supports Docker secrets for `DB_PASSWORD`. Instead of setting the password directly in `.env`, you can use:
168+
169+
```bash
170+
# Create a secrets file
171+
echo "my-secret-password" > backend/runtime/secrets/postgres_password_secret
172+
173+
# Set in .env or environment:
174+
DB_PASSWORD_FILE=/run/secrets/postgres_password_secret
175+
```
176+
177+
The entrypoint's `file_env` function reads the file and exports `DB_PASSWORD`. You cannot set both `DB_PASSWORD` and `DB_PASSWORD_FILE` — the entrypoint will error if both are present.
178+
179+
---
180+
181+
## Migration Notes
182+
183+
### From `DB_DEFAULT_*` to `DB_*` variables
184+
185+
Previous versions used `DB_DEFAULT_*` prefixed variables (e.g., `DB_DEFAULT_PASSWORD`, `DB_DEFAULT_HOST`). These have been replaced with `DB_*` variables (e.g., `DB_PASSWORD`, `DB_HOST`).
186+
187+
**If upgrading from a previous deployment:**
188+
189+
1. Rename variables in your `.env` / environment:
190+
- `DB_DEFAULT_PASSWORD` → `DB_PASSWORD`
191+
- `DB_DEFAULT_HOST` → `DB_HOST`
192+
- `DB_DEFAULT_PORT` → `DB_PORT`
193+
- `DB_DEFAULT_NAME` → `DB_NAME`
194+
- `DB_DEFAULT_USER` → `DB_USER`
195+
- `DB_DEFAULT_ENGINE` → `DB_ENGINE`
196+
197+
2. If using Docker secrets: rename `DB_DEFAULT_PASSWORD_FILE` → `DB_PASSWORD_FILE`.
198+
199+
3. The `DB_DEFAULT` JSON blob environment variable is no longer generated — `settings.py` reads individual `DB_*` variables directly.
200+
201+
---
202+
203+
## Related Documentation
204+
205+
- [Monitoring Setup](docs/monitoring.md) — Prometheus metrics configuration
206+
- [Notifications](docs/notifications.md) — Email and Discord notification setup

‎backend/setup-dashboard-db.sh‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
# Initializes dashboard database roles and databases via direct psql.
33
set -eu
44

5-
# Resolve env vars: prefer explicit names, fall back to backend naming (DB_DEFAULT_*)
5+
# Resolve env vars: prefer explicit names, fall back to backend naming (DB_*)
66
export DB_HOST="${DB_HOST:-dashboard_db}"
77
export DB_PORT="${DB_PORT:-5432}"
88
export DB_PASSWORD="${DB_PASSWORD:?DB_PASSWORD is required}"

‎backend/utils/docker/backend_entrypoint.sh‎

Lines changed: 1 addition & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ function file_env() {
2626
export "$var"="$val"
2727
}
2828

29-
file_env DB_DEFAULT_PASSWORD
29+
file_env DB_PASSWORD
3030

3131
# Initialize Prometheus metrics before Django starts
3232
PROMETHEUS_METRICS_ENABLED=$(echo "$PROMETHEUS_METRICS_ENABLED" | tr '[:upper:]' '[:lower:]')
@@ -44,19 +44,6 @@ if [ "$PROMETHEUS_METRICS_ENABLED" = "true" ]; then
4444
python3 utils/prometheus_aggregator.py &
4545
fi
4646

47-
export DB_DEFAULT="{
48-
\"ENGINE\": \"${DB_DEFAULT_ENGINE:=django.db.backends.postgresql}\",
49-
\"NAME\": \"${DB_DEFAULT_NAME:=dashboard}\",
50-
\"USER\": \"${DB_DEFAULT_USER:=admin}\",
51-
\"PASSWORD\": \"$DB_DEFAULT_PASSWORD\",
52-
\"HOST\": \"${DB_DEFAULT_HOST:=dashboard_db}\",
53-
\"PORT\": \"${DB_DEFAULT_PORT:=5432}\",
54-
\"CONN_MAX_AGE\": ${DB_DEFAULT_CONN_MAX_AGE:=null},
55-
\"OPTIONS\": {
56-
\"connect_timeout\": ${DB_DEFAULT_TIMEOUT:=16}
57-
}
58-
}"
59-
6047
chmod +x ./setup-dashboard-db.sh
6148
./setup-dashboard-db.sh
6249

0 commit comments

Comments
 (0)