|
| 1 | +.. _downstream-packaging: |
| 2 | + |
| 3 | +======================================== |
| 4 | +How to make downstream packaging easier? |
| 5 | +======================================== |
| 6 | + |
| 7 | +:Page Status: Draft |
| 8 | +:Last Reviewed: 2025-? |
| 9 | + |
| 10 | +While PyPI and the Python packaging tools such as :ref:`pip` are the primary |
| 11 | +means of distributing your packages, they are often also made available as part |
| 12 | +of other packaging ecosystems. These repackaging efforts are collectively called |
| 13 | +*downstream* packaging (your own efforts are called *upstream* packaging), |
| 14 | +and include such projects as Linux distributions, Conda, Homebrew and MacPorts. |
| 15 | +They often aim to provide good support for use cases that cannot be handled |
| 16 | +via Python packaging tools alone, such as good integration with non-Python |
| 17 | +software. |
| 18 | + |
| 19 | +This discussion attempts to explain how downstream packaging is usually done, |
| 20 | +and what challenges are downstream packagers facing. It ultimately aims to give |
| 21 | +you some hints on how you can make downstream packaging easier. |
| 22 | + |
| 23 | +Please note that downstream builds include not only binary redistribution, |
| 24 | +but also source builds done on user systems, in source-first distributions |
| 25 | +such as Gentoo Linux. |
| 26 | + |
| 27 | + |
| 28 | +.. _Provide complete source distributions: |
| 29 | + |
| 30 | +Provide complete source distributions |
| 31 | +------------------------------------- |
| 32 | +The vast majority of downstream packagers prefer to build packages from source, |
| 33 | +rather than use the upstream-provided binary packages. This is also true |
| 34 | +of pure Python packages that provide universal wheels. The reasons for using |
| 35 | +source distributions may include: |
| 36 | + |
| 37 | +- being able to audit the source code of all packages |
| 38 | + |
| 39 | +- being able to run the test suite and build documentation |
| 40 | + |
| 41 | +- being able to easily apply patches, including backporting commits from your |
| 42 | + repository and sending patches back to you |
| 43 | + |
| 44 | +- being able to build against a specific platform that is not covered |
| 45 | + by upstream builds |
| 46 | + |
| 47 | +- being able to build against specific versions of system libraries |
| 48 | + |
| 49 | +- having a consistent build process across all Python packages |
| 50 | + |
| 51 | +Ideally, a source distribution archive should include all the files necessary |
| 52 | +to build the package itself, run its test suite, build and install its |
| 53 | +documentation, and any other files that may be useful to end users, such |
| 54 | +as shell completions, editor support files, and so on. |
| 55 | + |
| 56 | +Some projects are concerned about increasing the size of source distribution, |
| 57 | +or do not wish Python packaging tools to fall back to source distributions |
| 58 | +automatically. In these cases, a good compromise may be to publish a separate |
| 59 | +source archive for downstream use, for example by attaching it to a GitHub |
| 60 | +release. |
| 61 | + |
| 62 | +While it is usually possible to build packages from a git repository, there are |
| 63 | +a few important reasons to provide a static archive file instead: |
| 64 | + |
| 65 | +- Fetching a single file is often more efficient, more reliable and better |
| 66 | + supported than e.g. using a git clone. This can help users with a shoddy |
| 67 | + Internet connection. |
| 68 | + |
| 69 | +- Downstreams often use checksums to verify the authenticity of source files |
| 70 | + on subsequent builds, which require that they remain bitwise identical over |
| 71 | + time. For example, automatically generated git archives do not guarantee |
| 72 | + that. |
| 73 | + |
| 74 | +- Archive files can be mirrored, reducing both upstream and downstream |
| 75 | + bandwidth use. The actual builds can afterwards be performed in firewalled |
| 76 | + or offline environments, that can only access source files provided |
| 77 | + by the local mirror or redistributed earlier. |
| 78 | + |
| 79 | +A good idea is to use a release workflow that starts by building a source |
| 80 | +distribution, and then performs all the remaining release steps (such as |
| 81 | +running tests and building wheels) from the unpacked source distribution. This |
| 82 | +ensures that the source distribution is actually tested, and reduces the risk |
| 83 | +that users installing from it will hit build failures or install an incomplete |
| 84 | +package. |
0 commit comments