Feature: new deploy setup (#1751)

* fix: correct default for trees file

If the ingester was initialized with an empty trees_file argument it was not defaulting correctly since None was passed to the function

* docker-compose: Add dashboard ingester as separate service

The ingester is used to insert data into the database, so we always want it running smoothly in the project
while also not consuming resources or having its resources consumed by the backend as if it were just a cron job

Also fixes signal handling when running the ingester command within docker
and avoids running cronjobs in the new service.

Co-authored-by: Marcelo Robert Santos <marcelo.santos@profusion.mobi>

* feat: add pending items processing as a docker service

Follows the addition of the ingester service. This new service will consume the items generated by the ingester aggregation and organize them into the listing tables. It is done as a new service instead of a cronjob in order to avoid over-execution and multiple processes being started

Also adds a dockerignore file to avoid building cache folder/files and the folder for submission testing

* docker-compose: Remove cloud-sql

Removed old configuration for Google Cloud because it has been replaced with Azure database

Co-authored-by: Marcelo Robert Santos <marcelo.santos@profusion.mobi>

* chore: remove gcloud from docs

* feat: update trees file with ingester startup

Replaces the use of a cronjob to update the tree-names file with updating it with the ingester. This also lets the command be called within the code.

* refactor: separate commands containers further

Gives each command their specific env file and also separates the backend entrypoint

* refactor: simplify database env vars

Reverts to using only a single set of variables for the database setup; Removes the json format of the database env vars

* chore: add dashboard database setup script and initialization (#1750)

* chore: add dashboard database setup script and initialization

* refactor: update database user and name variables in setup scripts

* fix: use correct image names in compose

Corrects the image names setting the right variables for the images that can be pulled from kernelci/dashboard

* refactor: consolidate database initialization into setup script

---------

Co-authored-by: Denys Fedoryshchenko <denys.f@collabora.com>
Co-authored-by: Gustavo Flores <gustavo.flores@profusion.mobi>

Author: 3 people

.env.backend.example

@@ -10,24 +10,21 @@ DEBUG_DB_VARS=${DEBUG_DB_VARS:-False}
 ENABLE_LOGGING=${ENABLE_LOGGING:-False} # Logs all prints
 DEBUG_SQL_QUERY=${DEBUG_SQL_QUERY:-False} # Logs the SQL queries made
 
-DB_DEFAULT_PASSWORD_FILE=/run/secrets/postgres_password_secret
-DB_DEFAULT_HOST=cloudsql-proxy
-DB_DEFAULT_USER=${DB_DEFAULT_USER:-kernelci}
-DB_DEFAULT_NAME=${DB_DEFAULT_NAME:-kcidb}
-
-DASH_DB_NAME=${DASH_DB_NAME:-dashboard}
-DASH_DB_USER=${DASH_DB_USER:-admin}
-DASH_DB_PASSWORD=${DASH_DB_PASSWORD:-admin}
-DASH_DB_HOST=dashboard_db
-DASH_DB_PORT=${DASH_DB_PORT:-5432}
-
-# Tells the backend to use DASH_DB as the default database
-USE_DASHBOARD_DB=${USE_DASHBOARD_DB:-False}
+DB_NAME=${DB_DEFAULT_NAME:-dashboard}
+DB_USER=${DB_DEFAULT_USER:-admin}
+DB_PASSWORD=db_password
+DB_HOST=dashboard_db  # Docker can't connect to the ssh tunnel host directly.
+DB_PORT=${DB_DEFAULT_PORT:-5432}
+DB_ENGINE=${DB_DEFAULT_ENGINE:-django.db.backends.postgresql}
+DB_OPTIONS_CONNECT_TIMEOUT=${DB_OPTIONS_CONNECT_TIMEOUT:-16}
 
 # Check docs/monitoring.md docs for more context
 PROMETHEUS_METRICS_ENABLED=False
 PROMETHEUS_METRICS_PORT=8001
 PROMETHEUS_MULTIPROC_DIR=/tmp/metrics
+INGESTER_METRICS_PORT=8002
+
+BACKEND_VOLUME_DIR=/volume_data
 
 ## Variables used for the notifications command. Check docs/notifications.md
 # EMAIL_HOST_USER="youruser@host" # (optional)

.env.db.example

@@ -1,3 +1,6 @@
-POSTGRES_USER=${DASH_DB_USER:-admin}
-POSTGRES_PASSWORD=${DASH_DB_PASSWORD:-admin}
-POSTGRES_DB=${DASH_DB_NAME:-dashboard}
+POSTGRES_USER=${DB_USER:-admin}
+POSTGRES_PASSWORD=${DB_PASSWORD:-admin}
+POSTGRES_DB=${DB_NAME:-dashboard}
+
+APP_DB_USER=${APP_DB_USER:-dashboard}
+APP_DB=${APP_DB:-dashboard}

.env.example

@@ -0,0 +1,4 @@
+# Top-level variables that should be used just for the docker-compose setup.
+INGESTER_METRICS_PORT=8002
+INGESTER_TREE_NAMES_FILE_DIR=../data
+INGESTER_SPOOL_DIR=../spool

.env.ingester.example

@@ -0,0 +1,19 @@
+DEBUG=${DEBUG:-False}
+DEBUG_DB_VARS=${DEBUG_DB_VARS:-False}
+ENABLE_LOGGING=${ENABLE_LOGGING:-False} # Logs all prints
+DEBUG_SQL_QUERY=${DEBUG_SQL_QUERY:-False} # Logs the SQL queries made
+
+DB_NAME=${DB_NAME:-dashboard}
+DB_USER=${DB_USER:-admin}
+DB_PASSWORD=db_password
+DB_HOST=${DB_HOST:-dashboard_db}  # Docker can't connect to the ssh tunnel host directly.
+DB_PORT=${DB_PORT:-5432}
+DB_ENGINE=${DB_ENGINE:-django.db.backends.postgresql}
+DB_OPTIONS_CONNECT_TIMEOUT=16
+
+BACKEND_VOLUME_DIR=/volume_data
+
+# Check docs/monitoring.md docs for more context
+PROMETHEUS_METRICS_ENABLED=False
+PROMETHEUS_METRICS_PORT=8001
+PROMETHEUS_MULTIPROC_DIR=/tmp/metrics

.env.pending_aggregations.example

@@ -0,0 +1,12 @@
+DEBUG=${DEBUG:-False}
+DEBUG_DB_VARS=${DEBUG_DB_VARS:-False}
+ENABLE_LOGGING=${ENABLE_LOGGING:-False} # Logs all prints
+DEBUG_SQL_QUERY=${DEBUG_SQL_QUERY:-False} # Logs the SQL queries made
+
+DB_NAME=${DB_NAME:-dashboard}
+DB_USER=${DB_USER:-admin}
+DB_PASSWORD=db_password
+DB_HOST=${DB_HOST:-dashboard_db}  # Docker can't connect to the ssh tunnel host directly.
+DB_PORT=${DB_PORT:-5432}
+DB_ENGINE=${DB_ENGINE:-django.db.backends.postgresql}
+DB_OPTIONS_CONNECT_TIMEOUT=16

.gitignore

@@ -3,7 +3,9 @@ application_default_credentials.json
 .vscode
 .env
 .env.backend
+.env.ingester
 .env.db
+.env.pending_aggregations
 .env.proxy
 .idea
 

README.md

@@ -53,7 +53,7 @@ pnpm preview
 
 Create a .env file in the base directory,
 ```sh
- cp .env.backend.example .env.bakckend
+ cp .env.backend.example .env.backend
 ```
 
 Create a secret key for Django:
@@ -62,32 +62,17 @@ export DJANGO_SECRET_KEY=$(openssl rand -base64 22)
 ```
 We are not using sessions or anything like that right now, so changing the secret key won't be a big deal.
 
+Since the production *database* is not open for the public, we use ssh tunneling with a whitelist to access it. This means that the docker setup currently can't access it, but we have a local database that is connected automatically if you don't change the env vars.
 
-Add a `application_default_credentials.json` file with your ADC in the root of the project.
-```sh
-gcloud auth application-default login
-cp ~/.config/gcloud/application_default_credentials.json .
-```
-**Important**: Check the `application_default_credentials.json` file permissions with `ls -l` to see if docker has access to it.
-
-After setting up your connection with Google Cloud with the following commands:
-
-```sh
-cloud-sql-proxy kernelci-production:us-central1:postgresql2 &
-gcloud auth application-default login
-```
-
- If it doesn't work, check the [Configure ADC with your Google Account](https://cloud.google.com/docs/authentication/provide-credentials-adc#google-idp) documentation.
-
-Create a secret file with the database password:
+If you do use docker, you should create a secret file with the database password:
 ```sh
 mkdir -p backend/runtime/secrets
 echo <password> > backend/runtime/secrets/postgres_password_secret
 ```
 
-If you are going to use a database user other than `kernelci`, set it to `DB_DEFAULT_USER`:
+If you are going to use a database user other than `kernelci`, set it to `DB_USER`:
 ```sh
-export DB_DEFAULT_USER=<user>
+export DB_USER=<user>
 ```
 
 If you are setting up instance different than production KernelCI dashboard, you need to define CORS_ALLOWED_ORIGINS. On .env.backend:
@@ -119,17 +104,15 @@ Or you can also run the env exports and docker compose within the root user by r
 > Tip: you can create a quick script to set all the necessary envs and start the services. This will also allow docker to see the environment variables correclty. Example:
 
 ```sh
-export DB_DEFAULT_USER=email@email.com
+export DB_USER=email@email.com
 export DJANGO_SECRET_KEY=$(openssl rand -base64 22)
-export DB_DEFAULT_NAME=kcidb
+export DB_NAME=kcidb
 export DISCORD_WEBHOOK_URL="https://discord.com/api/webhooks/..."
 
 docker compose up --build
 ```
 
-> [Note] If you are going to run using only the local database, the DB_DEFAULT_NAME should be `dashboard` and the `DB_DEFAULT_USER` and `DB_DEFAULT_PASSWORD` should be `admin` (for now).
-> After you define those values, also set the env var `USE_DASHBOARD_DB` to True, setting the local database as the default one.
-> You could also set the DB_DEFAULT variables to point to the local database and leave `USE_DASHBOARD_DB` as False.
+> [Note] If you are going to run using only the local database, the DB_NAME should be `dashboard` and the `DB_USER` and `DB_PASSWORD` should be `admin` (for now). This simply follows what is going to be setup by the `dashboard_db` service on docker compose.
 
 
 ## Deploying to production

backend/Dockerfile

@@ -6,6 +6,8 @@ FROM alpine:3.20 as backend-base
 WORKDIR /
 
 ENV POETRY_HOME=/opt/poetry
+ENV PYTHONOPTIMIZE=${PYTHONOPTIMIZE:-2}
+ENV PYTHONUNBUFFERED=${PYTHONUNBUFFERED:-1}
 
 RUN apk update \
     && apk add --no-cache \
@@ -21,10 +23,9 @@ RUN apk update \
     && $POETRY_HOME/bin/pip install poetry==1.8.3 \
     && ln -s $POETRY_HOME/bin/poetry /bin/poetry
 
-FROM backend-base
-COPY . /backend
-COPY ./utils/docker/entrypoint.sh /entrypoint.sh
+FROM backend-base as kernelci-django-backend
 WORKDIR /backend
+COPY . /backend
 
 ARG INSTALL_DEV_DEPS=false
 RUN if [ "$INSTALL_DEV_DEPS" = "true" ]; then \
@@ -36,20 +37,8 @@ RUN if [ "$INSTALL_DEV_DEPS" = "true" ]; then \
 ENV DJANGO_APP="kernelCI"
 
 # Precompile Python modules
-RUN O='0 1 2' \
-        PY_MAJMIN=`poetry run python -c "import sys; print('%s.%s'%sys.version_info[0:2])"` \
-        PY_D=`poetry env info --path` \
-        D="$PY_D $DJANGO_APP"; \
-        for N in $O; do \
-            echo "compile python $PY_MAJMIN byte code at -O$N: $D"; \
-            PYTHONOPTIMIZE=$N poetry run python -m compileall -q $D || exit 1; \
-        done
-
-# Expose both application and metrics ports
-EXPOSE 8000 8001
-
-# gunicorn is added here but is run at the end of the entrypoint.
-# This was done here such that we can run a custom command
-# in the backend container without using gunicorn
-CMD ["gunicorn", "kernelCI.wsgi:application", "--workers=5", "--forwarded-allow-ips=*", "--bind=0.0.0.0:8000", "--timeout=250"]
-ENTRYPOINT ["/entrypoint.sh"]
+RUN PY_MAJMIN=`poetry run python -c "import sys; print('%s.%s'%sys.version_info[0:2])"` \
+    PY_D=`poetry env info --path` \
+    D="$PY_D $DJANGO_APP"; \
+    echo "compile python $PY_MAJMIN byte code at $D"; \
+    poetry run python -m compileall -q $D || exit 1;

backend/README.md

@@ -41,44 +41,28 @@ It's possible to export `DEBUG_SQL_QUERY=True` if you want to see which SQL quer
 
 ### Databases
 
-For the main database, the backend uses a `DB_DEFAULT` variable that must have a JSON string such as:
+For the main database, the backend uses a series of `DB_` environment variables that have to be set such as:
 
 ```sh
-DB_DEFAULT="{
-    \"ENGINE\": \"${DB_DEFAULT_ENGINE:=django_prometheus.db.backends.postgresql}\",
-    \"NAME\": \"${DB_DEFAULT_NAME:=kcidb}\",
-    \"USER\": \"${DB_DEFAULT_USER:=<your-email-here>}\",
-    \"PASSWORD\": \"<your-password-here-don't-forget-to-scape-special-characters>\", 
-    \"HOST\": \"${DB_DEFAULT_HOST:=127.0.0.1}\",
-    \"PORT\": \"${DB_DEFAULT_PORT:=5432}\",
-    \"CONN_MAX_AGE\": ${DB_DEFAULT_CONN_MAX_AGE:=null},
-    \"OPTIONS\": {
-      \"connect_timeout\": ${DB_DEFAULT_TIMEOUT:=10}
-    }
-}"
+export DB_NAME=kcidb
+export DB_USER=your-email-here
+export DB_PASSWORD=your-password-here-dont-forget-to-scape-special-characters
+export DB_HOST=127.0.0.1
+export DB_PORT=5432
+export DB_ENGINE=django_prometheus.db.backends.postgresql
+export DB_OPTIONS_CONNECT_TIMEOUT=16
 ```
 > [!NOTE]
 > It is possible to have authentication issues when escaping special characters. In some cases, it is necessary to add more than one backslash, while in others, no addition is needed. To assist with this, you can export `DEBUG_DB_VARS=True` to check the database connection info in the terminal, allowing you to determine if the characters got escaped as intended. **This variable should NOT be set to True in production**.
 
-Along with the main database, the backend also connects to a secondary, local db made while we transition between database providers. It uses a simpler environment variable structure:
-```sh
-DASH_DB_NAME=dashboard
-DASH_DB_USER=admin
-DASH_DB_PASSWORD=admin
-DASH_DB_HOST=127.0.0.1
-DASH_DB_PORT=5434 # Note that this is not 5432, avoids conflict with the proxy port.
-```
-
-If you have both connections and you want to use the local one as the default, the env var `USE_DASHBOARD_DB` can be exported as `True` for a quick flag change.
-
 #### SQLite
 
 Before running the server, you must also update a local SQLite database used for caching. You can simply run the [migrate-cache-db.sh](./migrate-cache-db.sh) script, which will update the migrations if needed and apply them.
 
 
 ## Running the server
 
-After connecting to the database proxy, execute the server with:
+After connecting to the database, execute the server with:
 
 ```sh
 poetry run python3 manage.py runserver
@@ -89,7 +73,7 @@ poetry run python3 manage.py runserver
 We have a couple of useful scripts:
 
 * [migrate-cache-db.sh](./migrate-cache-db.sh) will create and apply migrations for the cache SQLite database. This runs automatically when running on docker, but you have to run it mannually otherwise.
-* [migrate-app-db.sh](./migrate-app-db.sh) will create and apply migrations for the main app, used for the secondary database. This does not run automatically in order to avoid potential problems. For this script, be aware of the `USE_DASHBOARD_DB` environment variable, because that variable will change the name of the databases (`dashboard_db` becomes `default` and `default` becomes `kcidb`) and so you'll have to change the command accordingly.
+* [migrate-app-db.sh](./migrate-app-db.sh) will create and apply migrations for the main app, generally used for the local database. This does not run automatically in order to avoid potential problems. Be aware that *if* you have write permissions to the production database it will be changed, so double check that you are connected to the right one (but usually developers won't have write access to the production database anyway).
 
 It is important to note that Django automatically creates migrations based on changes to the models when running the first command of the scripts above. You can edit the migrations manually, and you can also run the commands by hand if you want more control over it.
 
@@ -236,7 +220,7 @@ In order to debug backend in PyCharm, just follow these steps:
    - in `Run` session of the dialog, select `script`, then find the script `manage.py` at the `backend` folder
    - at `script` name input, just enter `runserver`
    - at `Environment Variables`, enter the following values:
-     - `DB_DEFAULT`: `{"ENGINE": "django.db.backends.postgresql", "NAME": "playground_kcidb", "USER": "<youremail>@profusion.mobi", "PASSWORD": "<yourpassword>", "HOST": "127.0.0.1", "PORT": "5432", "CONN_MAX_AGE": null, "OPTIONS": {"connect_timeout": 2, "sslmode": "disable"}}`
+     - `DB_ENGINE`: `django.db.backends.postgresql`, `DB_NAME`: `dashboard`, `DB_USER`: `<youremail>@profusion.mobi`, `DB_PASSWORD`: `<yourpassword>`, `DB_HOST`: `127.0.0.1`, `DB_PORT`: `5432`, `DB_OPTIONS_CONNECT_TIMEOUT`: `16`
      - `DEBUG`: `True`
 
 Quote character in password field is escaped normally with `\"` .

backend/data/notifications/example-subscription.yaml

@@ -1,4 +1,4 @@
-tree: # The filename and the name of the tree may follow the trees-name.yaml file, but does not matter in the code
+tree: # The filename and the name of the tree may follow the tree-names.yaml file, but does not matter in the code
   url: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git # the git_repository_url of the tree
   default_recipients:
     - laura.nao@collabora.com