add pdf artifact upload step (#3410)

* add pdf artifact upload step

* tried to fix artifact name

* add pdfs to ignore

* removed ignored pdfs

* artifacts downloaded in deploy script

* lint fixes

* shellcheck adjust

* adjusted descriptions in the script and readme

* Update src/README.md

* Update src/README.md

* Update src/README.md

Co-authored-by: Barry Pollard <barrypollard@google.com>

---------

Co-authored-by: Barry Pollard <barrypollard@google.com>

Author: max-ostapenko

.github/workflows/predeploy.yml

@@ -5,6 +5,7 @@
 # A pre-deploy script that must be run in GitHub Actions to:
 # - Update timestamps and hashes
 # - Generate the ebooks
+# - Upload ebooks as artifacts
 # - Submit a Pull Request with the changes
 #
 # This should be run before every release
@@ -63,6 +64,14 @@ jobs:
       run: |
         cd src
         npm run ebooks
+    - name: Upload PDF artifact
+      if: ${{ github.event.inputs.ebooks == 'true' }}
+      uses: actions/upload-artifact@v3
+      with:
+        name: pdfs
+        path: ./src/static/pdfs/*.pdf
+        if-no-files-found: warn # 'warn' or 'ignore' are also available, defaults to `warn`
+        retention-days: 5 # defaults to 90
     - name: Update timestamps
       run: |
         cd src
@@ -73,9 +82,9 @@ jobs:
       with:
         title: Pre-deploy Updates
         branch-suffix: timestamp
-        commit-message: Update Timestamps and Generate Ebooks
+        commit-message: Update Timestamps
         body: |
-          Updated Timestamps and Regenerated Ebooks through GitHub action
+          Updated Timestamps through GitHub action
           - Auto-generated by [create-pull-request][1] GitHub Action
 
           [1]: https://github.com/peter-evans/create-pull-request

.gitignore

@@ -4,3 +4,5 @@ env/
 *.log
 *.pyc
 **/.DS_Store
+src/static/pdfs/
+.env

src/README.md

@@ -177,7 +177,7 @@ It is possible to run the Super-Linter locally if you have Docker installed.
 First up pull the Super-Linter Docker image (this takes some time to download but only need to do this once or when you want to upgrade the version of the super-linter):
 
 ```
-docker pull github/super-linter:latest
+docker pull github/super-linter/slim:latest
 ```
 
 Then to run the linting do this:
@@ -253,7 +253,7 @@ To actually generate the ebooks, start your local server, then run the following
 npm run ebooks
 ```
 
-There is a GitHub Action which can be run manually from the Actions tab to generate the Ebooks and open a Pull Request for them. This is the easiest way to generate them.
+There is a GitHub Action which can be run manually from the Actions tab to generate the Ebooks and store the resulting files in an artifact. This is the easiest way to generate them.
 
 ## Generating ebooks - including print-ready ebooks if you want a hardcopy
 
@@ -291,7 +291,7 @@ This is the same as below since it uses all the default settings:
 prince "http://127.0.0.1:8080/en/2019/ebook?print&printer&page-size=A5&inside-margin=19.5mm&bleed=3mm&prince-trim=5mm&base-font-size=10px" -o static/pdfs/web_almanac_2019_en_print_A5.pdf
 ```
 
-Note this will create two extra pages at the begining which will need to be removed with a PDF editor (e.g. Adobe Acrobat) to start with a clean page starting on right hand side for printing. Please remove these before checking in versions into git.
+Note this will create two extra pages at the begining which will need to be removed with a PDF editor (e.g. Adobe Acrobat) to start with a clean page starting on right hand side for printing. Please remove these before deploying.
 
 It is also possible to generate a cover using the `&cover` URL param. This consists of basically 2 pages - the first page is a double width-page with front and back cover as one page (with spine in between) and the second page is a blank inside page.
 
@@ -332,6 +332,14 @@ You must first do some setup locally:
 gcloud init
 ```
 
+3. Set Github personal access token to GITHUB_TOKEN environment variable
+
+Use an existing token or create a new token [in GitHub.com](https://github.com/settings/personal-access-tokens/new). In token configuration select *Repository access = Public Repositories (read-only)*.
+
+```bash
+export GITHUB_TOKEN=...
+```
+
 ### Deploying staged versions of the site
 
 It's sometimes useful to deploy a staged version of the site for others to see this.
@@ -360,8 +368,9 @@ The deploy script will do the following:
 - Ask you to confirm you've run the pre-deploy script via GitHub Actions
 - Switch to the production branch
 - Merge changes from main
-- Do a clean install
-- Run the tests
+- Do a clean install (remove generated chapters and e-books)
+- Download latest e-books from Github Action artifacts
+- Run and test the website
 - Ask you to complete any local tests and confirm good to deploy
 - Ask for a version number (suggesing the last verision tagged and incrementing the patch)
 - Tag the release (after asking you for the version number to use)

src/tools/scripts/deploy.sh

@@ -1,8 +1,10 @@
 #!/bin/bash
 
 # This script is used to deploy the Web Almanac to Google Cloud Platform (GCP).
-# Users must have push permissions on the production branch and also release
-# permissions for the Web Almanac on GCP
+# Users must have the following permissions:
+# - push permissions on the production branch,
+# - Github permissions to download the latest e-books from Github Action artifacts,
+# - release permissions for the Web Almanac on GCP
 
 # exit when any command fails instead of trying to continue on
 set -e
@@ -98,10 +100,27 @@ if [ "$(pgrep -f 'python main.py')" ]; then
   pkill -9 python main.py
 fi
 
-#Remove generated chapters (in case new one from other branch in there)
+#Remove generated chapters and e-books (in case new one from other branch in there)
 # Or with "true" to prevent failure if files don't exist
-echo "Removing Generated Chapters"
+echo "Removing Generated Chapters and e-books"
 rm -f ./templates/*/*/chapters/* || true
+rm -f ./static/pdfs/* || true
+
+# Download latest e-books from Github Action artifacts
+echo "Downloading latest e-books from Github Action artifacts"
+# Set your GitHub repository and workflow name
+REPO_OWNER=HTTPArchive
+REPO_NAME=almanac.httparchive.org
+WORKFLOW_ID=predeploy.yml
+
+# Get the latest workflow run ID and download its artifact's ZIP file
+RUN_ID=$(curl -s -H "Authorization: token ${GITHUB_TOKEN}" "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/actions/workflows/${WORKFLOW_ID}/runs" | jq -r '.workflow_runs[0].id')
+ARTIFACT_ID=$(curl -s -H "Authorization: token ${GITHUB_TOKEN}" "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/actions/runs/${RUN_ID}/artifacts" | jq -r '.artifacts[0].id')
+curl -L -H "Authorization: token ${GITHUB_TOKEN}" -H "Accept: application/vnd.github.v3+json" "https://api.github.com/repos/${REPO_OWNER}/${REPO_NAME}/actions/artifacts/${ARTIFACT_ID}/zip" -o "${ARTIFACT_ID}.zip"
+
+# Extract the contents of the ZIP file and clean up
+unzip -q "${ARTIFACT_ID}.zip" -d ./static/pdfs/
+rm "${ARTIFACT_ID}.zip"
 
 echo "Run and test website"
 ./tools/scripts/run_and_test_website.sh