Merge branch 'main' into maintenance/intersphinx-setuptools

Author: di

.github/workflows/test.yml

@@ -0,0 +1,32 @@
+name: Test
+
+on: [push, pull_request]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: 3.9
+
+      - name: pip cache
+        uses: actions/cache@v2
+        with:
+          path: ~/.cache/pip
+          key:
+            ${{ matrix.os }}-${{ matrix.python-version }}-${{ hashFiles('**/requirements.txt') }}-${{ hashFiles('noxfile.py') }}
+          restore-keys: |
+            ${{ matrix.os }}-${{ matrix.python-version }}-
+
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade nox virtualenv
+
+      - name: Nox build
+        run: |
+          python -m nox -s build

.travis.yml

@@ -1,4 +1,5 @@
 sphinx==3.2.0
 sphinx-autobuild==0.7.1
+sphinx-inline-tabs==2020.10.19b4
 git+https://github.com/python/python-docs-theme.git#egg=python-docs-theme
 git+https://github.com/pypa/pypa-docs-theme.git#egg=pypa-docs-theme

requirements.txt

@@ -34,6 +34,7 @@
     'sphinx.ext.extlinks',
     'sphinx.ext.intersphinx',
     'sphinx.ext.todo',
+    'sphinx_inline_tabs',
 ]
 
 # Add any paths that contain templates here, relative to this directory.
@@ -196,8 +197,8 @@
 
 # Custom sidebar templates, filenames relative to this file.
 html_sidebars = {
-    '**': ['localtoc.html', 'relations.html'],
-    'index': ['localtoc.html']
+    '**': ['globaltoc.html', 'relations.html'],
+    'index': ['globaltoc.html']
 }
 
 # Additional templates that should be rendered to pages, maps page names to

source/conf.py

@@ -6,12 +6,12 @@ pip vs easy_install
 ===================
 
 
-`easy_install` was released in 2004, as part of :ref:`setuptools`.  It was
+:ref:`easy_install <easy_install>` was released in 2004, as part of :ref:`setuptools`.  It was
 notable at the time for installing :term:`packages <Distribution Package>` from
 :term:`PyPI <Python Package Index (PyPI)>` using requirement specifiers, and
 automatically installing dependencies.
 
-:ref:`pip` came later in 2008, as alternative to `easy_install`, although still
+:ref:`pip` came later in 2008, as alternative to :ref:`easy_install <easy_install>`, although still
 largely built on top of :ref:`setuptools` components.  It was notable at the
 time for *not* installing packages as :term:`Eggs <Egg>` or from :term:`Eggs <Egg>` (but
 rather simply as 'flat' packages from :term:`sdists <Source Distribution (or

source/discussions/pip-vs-easy-install.rst

@@ -44,7 +44,7 @@ Glossary
         A :term:`Built Distribution` format introduced by :ref:`setuptools`,
         which is being replaced by :term:`Wheel`.  For details, see `The
         Internal Structure of Python Eggs
-        <https://setuptools.readthedocs.io/en/latest/formats.html>`_ and
+        <https://setuptools.readthedocs.io/en/latest/deprecated/python_eggs.html>`_ and
         `Python Eggs <http://peak.telecommunity.com/DevCenter/PythonEggs>`_
 
     Extension Module

source/glossary.rst

@@ -68,35 +68,35 @@ the `BigQuery quickstart guide
 Data schema
 -----------
 
-Linehaul writes an entry in a ``the-psf.pypi.downloadsYYYYMMDD`` table for each
+Linehaul writes an entry in a ``the-psf.pypi.file_downloads`` table for each
 download. The table contains information about what file was downloaded and how
 it was downloaded. Some useful columns from the `table schema
-<https://console.cloud.google.com/bigquery?pli=1&p=the-psf&d=pypi&t=downloads&page=table>`__
+<https://console.cloud.google.com/bigquery?pli=1&p=the-psf&d=pypi&t=file_downloads&page=table>`__
 include:
 
-+------------------------+-----------------+-----------------------+
-| Column                 | Description     | Examples              |
-+========================+=================+=======================+
-| file.project           | Project name    | ``pipenv``, ``nose``  |
-+------------------------+-----------------+-----------------------+
-| file.version           | Package version | ``0.1.6``, ``1.4.2``  |
-+------------------------+-----------------+-----------------------+
-| details.installer.name | Installer       | pip, `bandersnatch`_  |
-+------------------------+-----------------+-----------------------+
-| details.python         | Python version  | ``2.7.12``, ``3.6.4`` |
-+------------------------+-----------------+-----------------------+
++------------------------+-----------------+-----------------------------+
+| Column                 | Description     | Examples                    |
++========================+=================+=============================+
+| timestamp              | Date and time   | ``2020-03-09 00:33:03 UTC`` |
++------------------------+-----------------+-----------------------------+
+| file.project           | Project name    | ``pipenv``, ``nose``        |
++------------------------+-----------------+-----------------------------+
+| file.version           | Package version | ``0.1.6``, ``1.4.2``        |
++------------------------+-----------------+-----------------------------+
+| details.installer.name | Installer       | pip, `bandersnatch`_        |
++------------------------+-----------------+-----------------------------+
+| details.python         | Python version  | ``2.7.12``, ``3.6.4``       |
++------------------------+-----------------+-----------------------------+
 
 
 Useful queries
 --------------
 
 Run queries in the `BigQuery web UI`_ by clicking the "Compose query" button.
 
-Note that the rows are stored in separate tables for each day, which helps
+Note that the rows are stored in a partitioned, which helps
 limit the cost of queries. These example queries analyze downloads from
-recent history by using `wildcard tables
-<https://cloud.google.com/bigquery/docs/querying-wildcard-tables>`__ to
-select all tables and then filter by ``_TABLE_SUFFIX``.
+recent history by filtering on the ``timestamp`` column.
 
 Counting package downloads
 ~~~~~~~~~~~~~~~~~~~~~~~~~~
@@ -108,18 +108,17 @@ The following query counts the total number of downloads for the project
 
     #standardSQL
     SELECT COUNT(*) AS num_downloads
-    FROM `the-psf.pypi.downloads*`
+    FROM `the-psf.pypi.file_downloads`
     WHERE file.project = 'pytest'
       -- Only query the last 30 days of history
-      AND _TABLE_SUFFIX
-        BETWEEN FORMAT_DATE(
-          '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
-        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
+      AND DATE(timestamp)
+        BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
+        AND CURRENT_DATE()
 
 +---------------+
 | num_downloads |
 +===============+
-| 2117807       |
+| 20531925      |
 +---------------+
 
 To only count downloads from pip, filter on the ``details.installer.name``
@@ -129,71 +128,94 @@ column.
 
     #standardSQL
     SELECT COUNT(*) AS num_downloads
-    FROM `the-psf.pypi.downloads*`
+    FROM `the-psf.pypi.file_downloads`
     WHERE file.project = 'pytest'
       AND details.installer.name = 'pip'
       -- Only query the last 30 days of history
-      AND _TABLE_SUFFIX
-        BETWEEN FORMAT_DATE(
-          '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
-        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
+      AND DATE(timestamp)
+        BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
+        AND CURRENT_DATE()
 
 +---------------+
 | num_downloads |
 +===============+
-| 1829322       |
+| 19391645      |
 +---------------+
 
 Package downloads over time
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To group by monthly downloads, use the ``_TABLE_SUFFIX`` pseudo-column. Also
-use the pseudo-column to limit the tables queried and the corresponding
-costs.
+To group by monthly downloads, use the ``TIMESTAMP_TRUNC`` function. Also
+filtering by this column reduces corresponding costs.
 
 ::
 
     #standardSQL
     SELECT
       COUNT(*) AS num_downloads,
-      SUBSTR(_TABLE_SUFFIX, 1, 6) AS `month`
-    FROM `the-psf.pypi.downloads*`
+      DATE_TRUNC(DATE(timestamp), MONTH) AS `month`
+    FROM `the-psf.pypi.file_downloads`
     WHERE
       file.project = 'pytest'
       -- Only query the last 6 months of history
-      AND _TABLE_SUFFIX
-        BETWEEN FORMAT_DATE(
-          '%Y%m01', DATE_SUB(CURRENT_DATE(), INTERVAL 6 MONTH))
-        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())
+      AND DATE(timestamp)
+        BETWEEN DATE_TRUNC(DATE_SUB(CURRENT_DATE(), INTERVAL 6 MONTH), MONTH)
+        AND CURRENT_DATE()
     GROUP BY `month`
     ORDER BY `month` DESC
 
-+---------------+--------+
-| num_downloads | month  |
-+===============+========+
-| 1956741       | 201801 |
-+---------------+--------+
-| 2344692       | 201712 |
-+---------------+--------+
-| 1730398       | 201711 |
-+---------------+--------+
-| 2047310       | 201710 |
-+---------------+--------+
-| 1744443       | 201709 |
-+---------------+--------+
-| 1916952       | 201708 |
-+---------------+--------+
-
-More queries
-~~~~~~~~~~~~
-
-- `Data driven decisions using PyPI download statistics
-  <https://langui.sh/2016/12/09/data-driven-decisions/>`__
-- `PyPI queries gist <https://gist.github.com/alex/4f100a9592b05e9b4d63>`__
-- `Python versions over time
-  <https://github.com/tswast/code-snippets/blob/master/2018/python-community-insights/Python%20Community%20Insights.ipynb>`__
-- `Non-Windows downloads, grouped by platform
-  <https://bigquery.cloud.google.com/savedquery/51422494423:ff1976af63614ad4a1258d8821dd7785>`__
++---------------+------------+
+| num_downloads | month      |
++===============+============+
+| 1956741       | 2018-01-01 |
++---------------+------------+
+| 2344692       | 2017-12-01 |
++---------------+------------+
+| 1730398       | 2017-11-01 |
++---------------+------------+
+| 2047310       | 2017-10-01 |
++---------------+------------+
+| 1744443       | 2017-09-01 |
++---------------+------------+
+| 1916952       | 2017-08-01 |
++---------------+------------+
+
+Python versions over time
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Extract the Python version from the ``details.python`` column. Warning: This
+query processes over 500 GB of data.
+
+::
+
+    #standardSQL
+    SELECT
+      REGEXP_EXTRACT(details.python, r"[0-9]+\.[0-9]+") AS python_version,
+      COUNT(*) AS num_downloads,
+    FROM `the-psf.pypi.file_downloads`
+    WHERE
+      -- Only query the last 6 months of history
+      DATE(timestamp)
+        BETWEEN DATE_TRUNC(DATE_SUB(CURRENT_DATE(), INTERVAL 6 MONTH), MONTH)
+        AND CURRENT_DATE()
+    GROUP BY `python_version`
+    ORDER BY `num_downloads` DESC
+
++--------+---------------+
+| python | num_downloads |
++========+===============+
+| 3.7    | 12990683561   |
++--------+---------------+
+| 3.6    | 9035598511    |
++--------+---------------+
+| 2.7    | 8467785320    |
++--------+---------------+
+| 3.8    | 4581627740    |
++--------+---------------+
+| 3.5    | 2412533601    |
++--------+---------------+
+| null   | 1641456718    |
++--------+---------------+
 
 Caveats
 =======
@@ -229,13 +251,12 @@ the official Python client library for BigQuery.
 
     query_job = client.query("""
     SELECT COUNT(*) AS num_downloads
-    FROM `the-psf.pypi.downloads*`
+    FROM `the-psf.pypi.file_downloads`
     WHERE file.project = 'pytest'
-    -- Only query the last 30 days of history
-    AND _TABLE_SUFFIX
-        BETWEEN FORMAT_DATE(
-            '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
-        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())""")
+      -- Only query the last 30 days of history
+      AND DATE(timestamp)
+        BETWEEN DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY)
+        AND CURRENT_DATE()""")
 
     results = query_job.result()  # Waits for job to complete.
     for row in results:

source/guides/analyzing-pypi-package-downloads.rst

@@ -508,9 +508,8 @@ entry_points
 Use this keyword to specify any plugins that your project provides for any named
 entry points that may be defined by your project or others that you depend on.
 
-For more information, see the section on `Dynamic Discovery of Services and
-Plugins
-<https://setuptools.readthedocs.io/en/latest/setuptools.html#dynamic-discovery-of-services-and-plugins>`_
+For more information, see the section on `Advertising Behavior
+<https://setuptools.readthedocs.io/en/latest/userguide/entry_point.html#dynamic-discovery-of-services-and-plugins>`_
 from the :ref:`setuptools` docs.
 
 The most commonly used entry point is "console_scripts" (see below).

source/guides/distributing-packages-using-setuptools.rst

@@ -13,28 +13,28 @@ jobs:
       uses: actions/setup-python@v1
       with:
         python-version: 3.7
-    - name: Install pep517
+    - name: Install pypa/build
       run: >-
         python -m
         pip install
-        pep517
+        build
         --user
     - name: Build a binary wheel and a source tarball
       run: >-
         python -m
-        pep517.build
-        --source
-        --binary
-        --out-dir dist/
+        build
+        --sdist
+        --wheel
+        --outdir dist/
         .
     # Actually publish to PyPI/TestPyPI
     - name: Publish distribution 📦 to Test PyPI
       uses: pypa/gh-action-pypi-publish@master
       with:
-        password: ${{ secrets.test_pypi_password }}
+        password: ${{ secrets.TEST_PYPI_API_TOKEN }}
         repository_url: https://test.pypi.org/legacy/
     - name: Publish distribution 📦 to PyPI
       if: startsWith(github.ref, 'refs/tags')
       uses: pypa/gh-action-pypi-publish@master
       with:
-        password: ${{ secrets.pypi_password }}
+        password: ${{ secrets.PYPI_API_TOKEN }}