Clarify specs for external installer interactions (#1215)

Co-authored-by: Paul Moore <p.f.moore@gmail.com>

Author: ncoghlan

source/specifications/direct-url.rst

@@ -6,8 +6,9 @@ Recording the Direct URL Origin of installed distributions
 ==========================================================
 
 This document specifies a :file:`direct_url.json` file in the
-:file:`*.dist-info` directory of an installed distribution, to record the
-Direct URL Origin of the distribution.
+``*.dist-info`` directory of an installed distribution, to record the
+Direct URL Origin of the distribution. The general structure and usage of
+``*.dist-info`` directories is described in :ref:`recording-installed-packages`.
 
 .. contents:: Contents
    :local:

source/specifications/entry-points.rst

@@ -81,8 +81,9 @@ File format
 
 Entry points are defined in a file called :file:`entry_points.txt` in the
 :file:`*.dist-info` directory of the distribution. This is the directory
-described in :pep:`376` for installed distributions, and in :pep:`427` for
-wheels.  The file uses the UTF-8 character encoding.
+described in :ref:`recording-installed-packages` for installed distributions,
+and in :ref:`binary-distribution-format` for wheels.
+The file uses the UTF-8 character encoding.
 
 The file contents are in INI format, as read by Python's :mod:`configparser`
 module. However, configparser treats names as case-insensitive by default,

source/specifications/externally-managed-environments.rst

@@ -0,0 +1,22 @@
+
+.. _externally-managed-environments:
+
+===============================
+Externally Managed Environments
+===============================
+
+While some Python installations are entirely managed by the user that installed
+Python, others may be provided and managed by another means (such as the
+operating system package manager in a Linux distribution, or as a bundled
+Python environment in an application with a dedicated installer).
+
+Attempting to use conventional Python packaging tools to manipulate such
+environments can be confusing at best and outright break the entire underlying
+operating system at worst. Documentation and interoperability guides only go
+so far in resolving such problems.
+
+:pep:`668` defined an ``EXTERNALLY-MANAGED`` marker file that allows a Python
+installation to indicate to Python-specific tools such as ``pip`` that they
+neither install nor remove packages into the interpreter’s default installation
+environment, and should instead guide the end user towards using
+:ref:`virtual-environments`.

source/specifications/index.rst

@@ -22,10 +22,18 @@ Package Distribution Metadata
    declaring-build-dependencies
    declaring-project-metadata
    platform-compatibility-tags
+
+Package Installation Environment Metadata
+-----------------------------------------
+
+.. toctree::
+   :maxdepth: 1
+
    recording-installed-packages
    entry-points
    direct-url
-
+   virtual-environments
+   externally-managed-environments
 
 Package Distribution File Formats
 ---------------------------------

source/specifications/recording-installed-packages.rst

@@ -21,12 +21,14 @@ History and change workflow
 ===========================
 
 The metadata described here was first specified in :pep:`376`, and later
-amended in :pep:`627`.
+amended in :pep:`627` (and other PEPs).
 It was formerly known as *Database of Installed Python Distributions*.
-Further amendments (except trivial language or typography fixes) must be made
-through the PEP process (see :pep:`1`).
+As with other PyPA specifications, editorial amendments with no functional
+impact may be made through the GitHub pull request workflow. Proposals for
+functional changes that would require amendments to package building and/or
+installation tools must be made through the PEP process (see :pep:`1`).
 
-While this document is the normative specification, these PEPs that introduce
+While this document is the normative specification, the PEPs that introduce
 changes to it may include additional information such as rationales and
 backwards compatibility considerations.
 
@@ -66,12 +68,14 @@ encouraged to start normalizing those fields.
     provide API for such tools to consume, so tools can have access to the
     unnormalized name when displaying distrubution information.
 
-This ``.dist-info`` directory can contain these files, described in detail
-below:
+This ``.dist-info`` directory may contain the following files, described in
+detail below:
 
 * ``METADATA``: contains project metadata
 * ``RECORD``: records the list of installed files.
 * ``INSTALLER``: records the name of the tool used to install the project.
+* ``entry_points.txt``: see :ref:`entry-points` for details
+* ``direct_url.json``: see :ref:`direct-url` for details
 
 The ``METADATA`` file is mandatory.
 All other files may be omitted at the installing tool's discretion.
@@ -169,7 +173,7 @@ Here is an example snippet of a possible ``RECORD`` file::
 
 If the ``RECORD`` file is missing, tools that rely on ``.dist-info`` must not
 attempt to uninstall or upgrade the package.
-(This does not apply to tools that rely on other sources of information,
+(This restriction does not apply to tools that rely on other sources of information,
 such as system package managers in Linux distros.)
 
 
@@ -197,6 +201,18 @@ For example, if a tool is asked to uninstall a project but finds no ``RECORD``
 file, it may suggest that the tool named in ``INSTALLER`` may be able to do the
 uninstallation.
 
+
+The entry_points.txt file
+=========================
+
+This file MAY be created by installers to indicate when packages contain
+components intended for discovery and use by other code, including console
+scripts and other applications that the installer has made available for
+execution.
+
+Its detailed specification is at :ref:`entry-points`.
+
+
 The direct_url.json file
 ========================
 
@@ -207,3 +223,32 @@ This file MUST NOT be created when installing a distribution from an other type
 of requirement (i.e. name plus version specifier).
 
 Its detailed specification is at :ref:`direct-url`.
+
+
+Intentionally preventing changes to installed packages
+======================================================
+
+In some cases (such as when needing to manage external dependencies in addition
+to Python ecosystem dependencies), it is desirable for a tool that installs
+packages into a Python environment to ensure that other tools are not used to
+uninstall or otherwise modify that installed package, as doing so may cause
+compatibility problems with the wider environment.
+
+To achieve this, affected tools should take the folllowing steps:
+
+* Rename or remove the ``RECORD`` file to prevent changes via other tools (e.g.
+  appending a suffix to create a non-standard ``RECORD.tool`` file if the tool
+  itself needs the information, or omitting the file entirely if the package
+  contents are tracked and managed via other means)
+* Write an ``INSTALLER`` file indicating the name of the tool that should be used
+  to manage the package (this allows ``RECORD``-aware tools to provide better
+  error notices when asked to modify affected packages)
+
+Python runtime providers may also prevent inadvertent modification of platform
+provided packages by modifying the default Python package installation scheme
+to use a location other than that used by platform provided packages (while also
+ensuring both locations appear on the default Python import path).
+
+In some circumstances, it may be desirable to block even installation of
+additional packages via Python-specific tools. For these cases refer to
+:ref:`externally-managed-environments`

source/specifications/virtual-environments.rst

@@ -0,0 +1,50 @@
+
+.. _virtual-environments:
+
+===========================
+Python Virtual Environments
+===========================
+
+For Python 3.3 and later versions, :pep:`405` introduced interpreter level support
+for the concept of "Python Virtual Environments". Each virtual environment has
+its own Python binary (allowing creation of environments with various Python
+versions) and can have its own independent set of installed Python packages in
+its site directories, but shares the standard library with the base installed
+Python. While the concept of virtual environments existed prior to this update,
+there was no previously standardised mechanism for declaring or discovering them.
+
+
+Runtime detection of virtual environments
+=========================================
+
+At runtime, virtual environments can be identified by virtue of ``sys.prefix``
+(the filesystem location of the running interpreter) having a different value
+from ``sys.base_prefix`` (the default filesytem location of the standard library
+directories).
+
+:ref:`venv-explanation` in the Python standard library documentation for the
+:py:mod:`venv` module covers this along with the concept of "activating" a
+virtual environment in an interactive operating system shell (this activation
+step is optional and hence the changes it makes can't be reliably used to
+detect whether a Python program is running in a virtual environment or not).
+
+
+Declaring installation environments as Python virtual environments
+==================================================================
+
+As described in :pep:`405`, a Python virtual environment in its simplest form
+consists of nothing more than a copy or symlink of the Python binary accompanied
+by a ``site-packages`` directory and a ``pyvenv.cfg`` file with a ``home`` key
+that indicates where to find the Python standard library modules.
+
+While designed to meet the needs of the standard :py:mod:`venv` module, this
+split installation and ``pyvenv.cfg`` file approach can be used by *any*
+Python installation provider that desires Python-specific tools to be aware that
+they are already operating in a virtual environment and no further environment
+nesting is required or desired.
+
+Even in the absence of a ``pyvenv.cfg`` file, any approach (e.g.
+``sitecustomize.py``, patching the installed Python runtime) that results in
+``sys.prefix`` and ``sys.base_prefix`` having different values, while still
+providing a matching default package installation scheme in ``sysconfig``, will
+be detected and behave as a Python virtual environment.