Merge branch 'master' into patch-1

Author: belm0

source/conf.py

@@ -53,7 +53,7 @@
 
 # General information about the project.
 project = u'Python Packaging User Guide'
-copyright = u'2013–2017, PyPA'
+copyright = u'2013–2019, PyPA'
 author = 'Python Packaging Authority'
 
 # The version info for the project you're documenting, acts as replacement for

source/guides/analyzing-pypi-package-downloads.rst

@@ -2,10 +2,10 @@
 Analyzing PyPI package downloads
 ================================
 
-This section covers how to use the `PyPI package dataset`_ to learn more
-about downloads of a package (or packages) hosted on PyPI. For example, you can
-use it to discover the distribution of Python versions used to download a
-package.
+This section covers how to use the public PyPI download statistics dataset
+to learn more about downloads of a package (or packages) hosted on PyPI. For
+example, you can use it to discover the distribution of Python versions used to
+download a package.
 
 .. contents:: Contents
    :local:
@@ -14,71 +14,45 @@ package.
 Background
 ==========
 
-PyPI does not display download statistics because they are difficult to
-collect and display accurately. Reasons for this are included in the
-`announcement email
-<https://mail.python.org/pipermail/distutils-sig/2013-May/020855.html>`__:
-
-    There are numerous reasons for [download counts] removal/deprecation some
-    of which are:
-
-        - Technically hard to make work with the new CDN
-
-            - The CDN is being donated to the PSF, and the donated tier does
-              not offer any form of log access
-            - The work around for not having log access would greatly reduce
-              the utility of the CDN
-        - Highly inaccurate
-            - A number of things prevent the download counts from being
-              inaccurate, some of which include:
-
-                - pip download cache
-                - Internal or unofficial mirrors
-                - Packages not hosted on PyPI (for comparisons sake)
-                - Mirrors or unofficial grab scripts causing inflated counts
-                  (Last I looked 25% of the downloads were from a known
-                  mirroring script).
-        - Not particularly useful
-
-            - Just because a project has been downloaded a lot doesn't mean
-              it's good
-            - Similarly just because a project hasn't been downloaded a lot
-              doesn't mean it's bad
-
-    In short because it's value is low for various reasons, and the tradeoffs
-    required to make it work are high It has been not an effective use of
-    resources.
-
-As an alternative, the `Linehaul project
-<https://github.com/pypa/linehaul>`__ streams download logs to `Google
-BigQuery`_ [#]_. Linehaul writes an entry in a
-``the-psf.pypi.downloadsYYYYMMDD`` table for each download. The table
-contains information about what file was downloaded and how it was
-downloaded. Some useful columns from the `table schema
-<https://bigquery.cloud.google.com/table/the-psf:pypi.downloads20161022?tab=schema>`__
-include:
+PyPI does not display download statistics for a number of reasons: [#]_
 
-+------------------------+-----------------+-----------------------+
-| Column                 | Description     | Examples              |
-+========================+=================+=======================+
-| file.project           | Project name    | ``pipenv``, ``nose``  |
-+------------------------+-----------------+-----------------------+
-| file.version           | Package version | ``0.1.6``, ``1.4.2``  |
-+------------------------+-----------------+-----------------------+
-| details.installer.name | Installer       | pip, `bandersnatch`_  |
-+------------------------+-----------------+-----------------------+
-| details.python         | Python version  | ``2.7.12``, ``3.6.4`` |
-+------------------------+-----------------+-----------------------+
+- **Inefficient to make work with a Content Distribution Network (CDN):**
+  Download statistics change constantly. Including them in project pages, which
+  are heavily cached, would require invalidating the cache more often, and
+  reduce the overall effectiveness of the cache.
 
-.. [#] `PyPI BigQuery dataset announcement email <https://mail.python.org/pipermail/distutils-sig/2016-May/028986.html>`__
+- **Highly inaccurate:** A number of things prevent the download counts from
+  being accurate, some of which include:
 
-Setting up
-==========
+  - ``pip``'s download cache (lowers download counts)
+  - Internal or unofficial mirrors (can both raise or lower download counts)
+  - Packages not hosted on PyPI (for comparisons sake)
+  - Unofficial scripts or attempts at download count inflation (raises download
+    counts)
+  - Known historical data quality issues (lowers download counts)
+
+- **Not particularly useful:** Just because a project has been downloaded a lot
+  doesn't mean it's good; Similarly just because a project hasn't been
+  downloaded a lot doesn't mean it's bad!
 
-In order to use `Google BigQuery`_ to query the `PyPI package dataset`_,
-you'll need a Google account and to enable the BigQuery API on a Google
-Cloud Platform project. You can run the up to 1TB of queries per month `using
-the BigQuery free tier without a credit card
+In short, because it's value is low for various reasons, and the tradeoffs
+required to make it work are high, it has been not an effective use of
+limited resources.
+
+Public dataset
+==============
+
+As an alternative, the `Linehaul project <https://github.com/pypa/linehaul>`__
+streams download logs from PyPI to `Google BigQuery`_ [#]_, where they are
+stored as a public dataset.
+
+Getting set up
+--------------
+
+In order to use `Google BigQuery`_ to query the `public PyPI download
+statistics dataset`_, you'll need a Google account and to enable the BigQuery
+API on a Google Cloud Platform project. You can run the up to 1TB of queries
+per month `using the BigQuery free tier without a credit card
 <https://cloud.google.com/blog/big-data/2017/01/how-to-run-a-terabyte-of-google-bigquery-queries-each-month-without-a-credit-card>`__
 
 - Navigate to the `BigQuery web UI`_.
@@ -90,8 +64,31 @@ For more detailed instructions on how to get started with BigQuery, check out
 the `BigQuery quickstart guide
 <https://cloud.google.com/bigquery/docs/quickstarts/quickstart-web-ui>`__.
 
+
+Data schema
+-----------
+
+Linehaul writes an entry in a ``the-psf.pypi.downloadsYYYYMMDD`` table for each
+download. The table contains information about what file was downloaded and how
+it was downloaded. Some useful columns from the `table schema
+<https://console.cloud.google.com/bigquery?pli=1&p=the-psf&d=pypi&t=downloads&page=table>`__
+include:
+
++------------------------+-----------------+-----------------------+
+| Column                 | Description     | Examples              |
++========================+=================+=======================+
+| file.project           | Project name    | ``pipenv``, ``nose``  |
++------------------------+-----------------+-----------------------+
+| file.version           | Package version | ``0.1.6``, ``1.4.2``  |
++------------------------+-----------------+-----------------------+
+| details.installer.name | Installer       | pip, `bandersnatch`_  |
++------------------------+-----------------+-----------------------+
+| details.python         | Python version  | ``2.7.12``, ``3.6.4`` |
++------------------------+-----------------+-----------------------+
+
+
 Useful queries
-==============
+--------------
 
 Run queries in the `BigQuery web UI`_ by clicking the "Compose query" button.
 
@@ -102,7 +99,7 @@ recent history by using `wildcard tables
 select all tables and then filter by ``_TABLE_SUFFIX``.
 
 Counting package downloads
---------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 The following query counts the total number of downloads for the project
 "pytest".
@@ -148,7 +145,7 @@ column.
 +---------------+
 
 Package downloads over time
----------------------------
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
 To group by monthly downloads, use the ``_TABLE_SUFFIX`` pseudo-column. Also
 use the pseudo-column to limit the tables queried and the corresponding
@@ -188,7 +185,7 @@ costs.
 +---------------+--------+
 
 More queries
-------------
+~~~~~~~~~~~~
 
 - `Data driven decisions using PyPI download statistics
   <https://langui.sh/2016/12/09/data-driven-decisions/>`__
@@ -198,19 +195,68 @@ More queries
 - `Non-Windows downloads, grouped by platform
   <https://bigquery.cloud.google.com/savedquery/51422494423:ff1976af63614ad4a1258d8821dd7785>`__
 
+Caveats
+=======
+
+In addition to the caveats listed in the background above, Linehaul suffered
+from a bug which caused it to significantly under-report download statistics
+prior to July 26, 2018. Downloads before this date are proportionally accurate
+(e.g. the percentage of Python 2 vs. Python 3 downloads) but total numbers are
+lower than actual by an order of magnitude.
+
+
 Additional tools
 ================
 
-You can also access the `PyPI package dataset`_ programmatically via the
-BigQuery API.
+Besides using the BigQuery console, there are some additional tools which may
+be useful when analyzing download statistics.
+
+``google-cloud-bigquery``
+-------------------------
 
-pypinfo
--------
+You can also access the public PyPI download statistics dataset
+programmatically via the BigQuery API and the `google-cloud-bigquery`_ project,
+the official Python client library for BigQuery.
+
+.. code-block:: python
+
+    from google.cloud import bigquery
+
+    # Note: depending on where this code is being run, you may require
+    # additional authentication. See:
+    # https://cloud.google.com/bigquery/docs/authentication/
+    client = bigquery.Client()
+
+    query_job = client.query("""
+    SELECT COUNT(*) AS num_downloads
+    FROM `the-psf.pypi.downloads*`
+    WHERE file.project = 'pytest'
+    -- Only query the last 30 days of history
+    AND _TABLE_SUFFIX
+        BETWEEN FORMAT_DATE(
+            '%Y%m%d', DATE_SUB(CURRENT_DATE(), INTERVAL 30 DAY))
+        AND FORMAT_DATE('%Y%m%d', CURRENT_DATE())""")
+
+    results = query_job.result()  # Waits for job to complete.
+    for row in results:
+        print("{} downloads".format(row.num_downloads))
+
+
+``pypinfo``
+-----------
 
 `pypinfo`_ is a command-line tool which provides access to the dataset and
 can generate several useful queries. For example, you can query the total
 number of download for a package with the command ``pypinfo package_name``.
 
+Install `pypinfo`_ using pip.
+
+::
+
+    pip install pypinfo
+
+Usage:
+
 ::
 
     $ pypinfo requests
@@ -223,20 +269,20 @@ number of download for a package with the command ``pypinfo package_name``.
     | -------------- |
     |      9,316,415 |
 
-Install `pypinfo`_ using pip.
 
-::
+``pandas-gbq``
+--------------
+
+The `pandas-gbq`_ project allows for accessing query results via `Pandas`_.
 
-    pip install pypinfo
 
-Other libraries
----------------
+References
+==========
 
-- `google-cloud-bigquery`_ is the official client library to access the
-  BigQuery API.
-- `pandas-gbq`_ allows for accessing query results via `Pandas`_.
+.. [#] `PyPI Download Counts deprecation email <https://mail.python.org/pipermail/distutils-sig/2013-May/020855.html>`__
+.. [#] `PyPI BigQuery dataset announcement email <https://mail.python.org/pipermail/distutils-sig/2016-May/028986.html>`__
 
-.. _PyPI package dataset: https://bigquery.cloud.google.com/dataset/the-psf:pypi
+.. _public PyPI download statistics dataset: https://console.cloud.google.com/bigquery?p=the-psf&d=pypi&page=dataset
 .. _bandersnatch: /key_projects/#bandersnatch
 .. _Google BigQuery: https://cloud.google.com/bigquery
 .. _BigQuery web UI: https://console.cloud.google.com/bigquery

source/guides/distributing-packages-using-setuptools.rst

@@ -353,13 +353,14 @@ packages
 
 ::
 
-  packages=find_packages(exclude=['contrib', 'docs', 'tests*']),
+  packages=find_packages(include=['sample', 'sample.*']),
 
 Set ``packages`` to a list of all :term:`packages <Import Package>` in your
 project, including their subpackages, sub-subpackages, etc.  Although the
 packages can be listed manually, ``setuptools.find_packages()`` finds them
-automatically.  Use the ``exclude`` keyword argument to omit packages that are
-not intended to be released and installed.
+automatically.  Use the ``include`` keyword argument to find only the given
+packages.  Use the ``exclude`` keyword argument to omit packages that are not
+intended to be released and installed.
 
 
 py_modules

source/guides/github-actions-ci-cd-sample/publish-to-test-pypi.yml

@@ -0,0 +1,40 @@
+name: Publish Python 🐍 distributions 📦 to PyPI and TestPyPI
+
+on: push
+
+jobs:
+  build-n-publish:
+    name: Build and publish Python 🐍 distributions 📦 to PyPI and TestPyPI
+    runs-on: ubuntu-18.04
+
+    steps:
+    - uses: actions/checkout@master
+    - name: Set up Python 3.7
+      uses: actions/setup-python@v1
+      with:
+        python-version: 3.7
+    - name: Install pep517
+      run: >-
+        python -m
+        pip install
+        pep517
+        --user
+    - name: Build a binary wheel and a source tarball
+      run: >-
+        python -m
+        pep517.build
+        --source
+        --binary
+        --out-dir dist/
+        .
+    # Actually publish to PyPI/TestPyPI
+    - name: Publish distribution 📦 to Test PyPI
+      uses: pypa/gh-action-pypi-publish@master
+      with:
+        password: ${{ secrets.test_pypi_password }}
+        repository_url: https://test.pypi.org/legacy/
+    - name: Publish distribution 📦 to PyPI
+      if: startsWith(github.event.ref, 'refs/tags')
+      uses: pypa/gh-action-pypi-publish@master
+      with:
+        password: ${{ secrets.pypi_password }}

source/guides/index-mirrors-and-caches.rst

@@ -30,8 +30,8 @@ pip provides a number of facilities for speeding up installation by using local
 cached copies of :term:`packages <Distribution Package>`:
 
 1. `Fast & local installs
-   <https://pip.pypa.io/en/latest/user_guide.html#fast-local-installs>`_ by
-   downloading all the requirements for a project and then pointing pip at
+   <https://pip.pypa.io/en/latest/user_guide/#installing-from-local-packages>`_
+   by downloading all the requirements for a project and then pointing pip at
    those downloaded files instead of going to PyPI.
 2. A variation on the above which pre-builds the installation files for
    the requirements using `pip wheel

source/guides/index.rst

@@ -29,3 +29,4 @@ introduction to packaging, see :doc:`/tutorials/index`.
    migrating-to-pypi-org
    using-testpypi
    making-a-pypi-friendly-readme
+   publishing-package-distribution-releases-using-github-actions-ci-cd-workflows

source/guides/installing-stand-alone-command-line-tools.rst

@@ -24,8 +24,8 @@ packages, and allows you to safely run the program from anywhere.
 
 ::
 
-  $ pip install --user pipx
-  $ pipx ensurepath  # ensures the path of the CLI application directory is on your $PATH
+  $ python3 -m pip install --user pipx
+  $ python3 -m pipx ensurepath  # ensures the path of the CLI application directory is on your $PATH
 
 .. Note:: You may need to restart your terminal for the path updates to take effect.
 

source/guides/installing-using-pip-and-virtual-environments.rst

@@ -184,7 +184,7 @@ Installing packages
 -------------------
 
 Now that you're in your virtual environment you can install packages. Let's install the
-excellent `Requests`_ library from the :term:`Python Package Index (PyPI)`:
+`Requests`_ library from the :term:`Python Package Index (PyPI)`:
 
 .. code-block:: bash
 
@@ -207,7 +207,7 @@ pip should download requests and all of its dependencies and install them:
     Installing collected packages: chardet, urllib3, certifi, idna, requests
     Successfully installed certifi-2017.7.27.1 chardet-3.0.4 idna-2.6 requests-2.18.4 urllib3-1.22
 
-.. _Requests: http://docs.python-requests.org/
+.. _Requests: https://pypi.org/project/requests/
 
 
 Installing specific versions