Merge pull request #1379 from chrysle/update-namespace-packaging-guide

Improve namespace packaging guide

Author: willingc

source/discussions/src-layout-vs-flat-layout.rst

@@ -1,3 +1,5 @@
+.. _src-layout-vs-flat-layout:
+
 =========================
 src layout vs flat layout
 =========================

source/guides/packaging-namespace-packages.rst

@@ -19,7 +19,7 @@ have the following package structure:
             __init__.py
             ...
         module_b.py
-    setup.py
+    pyproject.toml
 
 And you use this package in your code like so::
 
@@ -31,17 +31,19 @@ Then you can break these sub-packages into two separate distributions:
 .. code-block:: text
 
     mynamespace-subpackage-a/
-        setup.py
-        mynamespace/
-            subpackage_a/
-                __init__.py
+        pyproject.toml
+        src/
+            mynamespace/
+                subpackage_a/
+                    __init__.py
 
     mynamespace-subpackage-b/
-        setup.py
-        mynamespace/
-            subpackage_b/
-                __init__.py
-            module_b.py
+        pyproject.toml
+        src/
+            mynamespace/
+                subpackage_b/
+                    __init__.py
+                module_b.py
 
 Each sub-package can now be separately installed, used, and versioned.
 
@@ -57,52 +59,72 @@ import object short).
 Creating a namespace package
 ============================
 
-There are currently three different approaches to creating namespace packages:
+There are currently two different approaches to creating namespace packages,
+from which the latter is discouraged:
 
 #. Use `native namespace packages`_. This type of namespace package is defined
    in :pep:`420` and is available in Python 3.3 and later. This is recommended if
    packages in your namespace only ever need to support Python 3 and
    installation via ``pip``.
-#. Use `pkgutil-style namespace packages`_. This is recommended for new
-   packages that need to support Python 2 and 3 and installation via both
-   ``pip`` and ``python setup.py install``.
-#. Use `pkg_resources-style namespace packages`_. This method is recommended if
-   you need compatibility with packages already using this method or if your
-   package needs to be zip-safe.
-
-.. warning:: While native namespace packages and pkgutil-style namespace
-    packages are largely compatible, pkg_resources-style namespace packages
-    are not compatible with the other methods. It's inadvisable to use
-    different methods in different distributions that provide packages to the
-    same namespace.
+#. Use `legacy namespace packages`_. This comprises `pkgutil-style namespace packages`_
+   and `pkg_resources-style namespace packages`_.
 
 Native namespace packages
 -------------------------
 
 Python 3.3 added **implicit** namespace packages from :pep:`420`. All that is
 required to create a native namespace package is that you just omit
 :file:`__init__.py` from the namespace package directory. An example file
-structure:
+structure (following :ref:`src-layout <setuptools:src-layout>`):
 
 .. code-block:: text
 
-    setup.py
-    mynamespace/
-        # No __init__.py here.
-        subpackage_a/
-            # Sub-packages have __init__.py.
-            __init__.py
-            module.py
+    mynamespace-subpackage-a/
+        pyproject.toml # AND/OR setup.py, setup.cfg
+        src/
+            mynamespace/ # namespace package
+                # No __init__.py here.
+                subpackage_a/
+                    # Regular import packages have an __init__.py.
+                    __init__.py
+                    module.py
 
 It is extremely important that every distribution that uses the namespace
 package omits the :file:`__init__.py` or uses a pkgutil-style
 :file:`__init__.py`. If any distribution does not, it will cause the namespace
 logic to fail and the other sub-packages will not be importable.
 
-Because ``mynamespace`` doesn't contain an :file:`__init__.py`,
-:func:`setuptools.find_packages` won't find the sub-package. You must
-use :func:`setuptools.find_namespace_packages` instead or explicitly
-list all packages in your :file:`setup.py`. For example:
+The ``src-layout`` directory structure allows automatic discovery of packages
+by most :term:`build backends <Build Backend>`. See :ref:`src-layout-vs-flat-layout`
+for more information. If however you want to manage exclusions or inclusions of packages
+yourself, this is possible to be configured in the top-level :file:`pyproject.toml`:
+
+.. code-block:: toml
+
+    [build-system]
+    ...
+
+    [tool.setuptools.packages.find]
+    where = ["src/"]
+    include = ["mynamespace.subpackage_a"]
+
+    [project]
+    name = "mynamespace-subpackage-a"
+    ...
+
+The same can be accomplished with a :file:`setup.cfg`:
+
+.. code-block:: ini
+
+    [options]
+    package_dir =
+        =src
+    packages = find_namespace:
+
+    [options.packages.find]
+    where = src
+
+Or :file:`setup.py`:
 
 .. code-block:: python
 
@@ -111,9 +133,13 @@ list all packages in your :file:`setup.py`. For example:
     setup(
         name='mynamespace-subpackage-a',
         ...
-        packages=find_namespace_packages(include=['mynamespace.*'])
+        packages=find_namespace_packages(where='src/', include=['mynamespace.subpackage_a']),
+        package_dir={'': 'src'},
     )
 
+:ref:`setuptools` will search the directory structure for implicit namespace
+packages by default.
+
 A complete working example of two native namespace packages can be found in
 the `native namespace package example project`_.
 
@@ -125,8 +151,25 @@ the `native namespace package example project`_.
     only support Python 3 and pkgutil-style namespace packages in the
     distributions that need to support Python 2 and 3.
 
+
+Legacy namespace packages
+-------------------------
+
+These two methods, that were used to create namespace packages prior to :pep:`420`,
+are now considered to be obsolete and should not be used unless you need compatibility
+with packages already using this method. Also, :doc:`pkg_resources <setuptools:pkg_resources>`
+has been deprecated.
+
+To migrate an existing package, all packages sharing the namespace must be migrated simultaneously.
+
+.. warning:: While native namespace packages and pkgutil-style namespace
+    packages are largely compatible, pkg_resources-style namespace packages
+    are not compatible with the other methods. It's inadvisable to use
+    different methods in different distributions that provide packages to the
+    same namespace.
+
 pkgutil-style namespace packages
---------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 Python 2.3 introduced the :doc:`pkgutil <python:library/pkgutil>` module and the
 :py:func:`python:pkgutil.extend_path` function. This can be used to declare namespace
@@ -138,22 +181,24 @@ To create a pkgutil-style namespace package, you need to provide an
 
 .. code-block:: text
 
-    setup.py
-    mynamespace/
-        __init__.py  # Namespace package __init__.py
-        subpackage_a/
-            __init__.py  # Sub-package __init__.py
-            module.py
+    mynamespace-subpackage-a/
+        src/
+            pyproject.toml # AND/OR setup.cfg, setup.py
+            mynamespace/
+                __init__.py  # Namespace package __init__.py
+                subpackage_a/
+                    __init__.py  # Regular package __init__.py
+                    module.py
 
 The :file:`__init__.py` file for the namespace package needs to contain
-**only** the following:
+the following:
 
 .. code-block:: python
 
     __path__ = __import__('pkgutil').extend_path(__path__, __name__)
 
-**Every** distribution that uses the namespace package must include an
-identical :file:`__init__.py`. If any distribution does not, it will cause the
+**Every** distribution that uses the namespace package must include such
+an :file:`__init__.py`. If any distribution does not, it will cause the
 namespace logic to fail and the other sub-packages will not be importable.  Any
 additional code in :file:`__init__.py` will be inaccessible.
 
@@ -167,7 +212,7 @@ in the `pkgutil namespace example project`_.
 
 
 pkg_resources-style namespace packages
---------------------------------------
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 :doc:`Setuptools <setuptools:index>` provides the `pkg_resources.declare_namespace`_ function and
 the ``namespace_packages`` argument to :func:`~setuptools.setup`. Together
@@ -183,22 +228,24 @@ To create a pkg_resources-style namespace package, you need to provide an
 
 .. code-block:: text
 
-    setup.py
-    mynamespace/
-        __init__.py  # Namespace package __init__.py
-        subpackage_a/
-            __init__.py  # Sub-package __init__.py
-            module.py
+    mynamespace-subpackage-a/
+        src/
+            pyproject.toml # AND/OR setup.cfg, setup.py
+            mynamespace/
+                __init__.py  # Namespace package __init__.py
+                subpackage_a/
+                    __init__.py  # Regular package __init__.py
+                    module.py
 
 The :file:`__init__.py` file for the namespace package needs to contain
-**only** the following:
+the following:
 
 .. code-block:: python
 
     __import__('pkg_resources').declare_namespace(__name__)
 
-**Every** distribution that uses the namespace package must include an
-identical :file:`__init__.py`. If any distribution does not, it will cause the
+**Every** distribution that uses the namespace package must include such an
+:file:`__init__.py`. If any distribution does not, it will cause the
 namespace logic to fail and the other sub-packages will not be importable.  Any
 additional code in :file:`__init__.py` will be inaccessible.