Packaging & Tooling

Python packaging has a bad reputation. That reputation is earned — but it’s improving fast. Most of the confusion comes from a decade of competing standards that all still work. This chapter cuts through it.

The Packaging Ecosystem — A Historical Mess

Python shipped with distutils in 2000. distutils was enough to install pure Python packages but broke the moment you needed C extensions or complex builds. setuptools (2004) patched the gaps and added easy_install and eggs. Then pip (2008) replaced easy_install. Then conda showed up for scientific computing. Then poetry, pdm, hatch, flit, uv…

The core problem: packaging was bolted onto Python after the fact, and every tool that fixed one pain point created another. setup.py mixed configuration with executable code, which broke reproducibility. You couldn’t install a package without running arbitrary code.

PEP 517/518 (2017–2018) finally broke the cycle. PEP 518 introduced pyproject.toml as a standard config file. PEP 517 defined a build system interface — now a build backend (setuptools, hatchling, etc.) must expose standard hooks, and a build frontend (pip, build) calls those hooks. No more python setup.py install.

PEP 621 (2020) standardized the [project] table inside pyproject.toml, so every tool that reads project metadata reads the same format.

ELI5: Imagine every restaurant used a different order form — one used napkins, one used chalkboards, one used apps. PEP 517/518 said: everyone uses the same digital form. PEP 621 said: the form must have name, price, and allergens in the same spots on every restaurant’s app.

The result: setup.py is legacy. setup.cfg is legacy. pyproject.toml is the standard. If you’re starting a new project today, you don’t touch setup.py at all.

pyproject.toml — The Modern Standard

Three main tables you care about.

[build-system]

Tells pip and other frontends which backend to use to build your package:

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

Without this table, pip falls back to setuptools. Always specify it explicitly.

[project]

Project metadata per PEP 621:

[project]
name = "my-library"
version = "1.2.0"
description = "Does something useful"
requires-python = ">=3.11"
dependencies = [
    "httpx>=0.27",
    "pydantic>=2.0",
]

[project.optional-dependencies]
dev = ["pytest>=8", "ruff", "mypy"]
docs = ["mkdocs-material"]

[project.scripts]
my-cli = "my_library.cli:main"

optional-dependencies replaces extras_require. Install with pip install my-library[dev].

[tool.*]

Every tool that can read pyproject.toml gets a namespace here:

[tool.ruff]
line-length = 88
target-version = "py311"

[tool.mypy]
strict = true
python_version = "3.11"

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-ra -q"

This replaces .flake8, mypy.ini, pytest.ini, tox.ini, .isort.cfg, etc. One file.

Build Backend Comparison

Rule of thumb: new project → hatchling. Rust extension → maturin. Simple library, minimal deps → flit. Existing project → stay on setuptools unless you have a reason to migrate.

ELI5: The build backend is the kitchen. The build frontend (pip, uv, build) is the waiter. PEP 517 is the standard that says “the waiter asks for food using the same menu format regardless of which kitchen is in the back.”

Virtual Environments

Python packages install globally by default. Global installs break when project A needs requests==2.28 and project B needs requests==2.32. Virtual environments solve this by giving each project its own isolated copy of Python and site-packages.

ELI5: A virtual environment is a snapshot of a Python installation that only your project sees. It’s like each project gets its own apartment with its own pantry, instead of everyone sharing one kitchen.

venv vs virtualenv

venv is almost always enough. Use virtualenv only if you need its specific features (older Python support, seed packages, faster creation in tight loops).

How They Work

.venv/
  bin/
    python3      → symlink or copy of the Python binary
    pip
    activate     → shell script that sets PATH and VIRTUAL_ENV
  lib/
    python3.11/
      site-packages/   → your installed packages go here
  pyvenv.cfg           → records base interpreter path

When you activate (source .venv/bin/activate), it prepends .venv/bin to your PATH and sets VIRTUAL_ENV. The Python binary inside .venv has sys.prefix pointing to .venv/, so site-packages resolves to the local one.

Activating is optional. ./venv/bin/python script.py and ./venv/bin/pip install work fine without activating. CI pipelines usually skip activation entirely.

Common mistake: Committing .venv/ to git. Always add it to .gitignore. The environment is reproducible from requirements.txt or pyproject.toml.

Modern Package Managers

pip

The baseline. Talks to PyPI, installs packages. Problems: no real lockfile (requirements.txt is manual), the resolver was famously slow and buggy until 2020, no built-in way to distinguish direct vs transitive dependencies.

Still the right tool when: you need maximum compatibility, you’re in a constrained environment, or you’re chaining with pip-tools.

pip-tools

Two commands: pip-compile reads requirements.in (or pyproject.toml) and produces a fully pinned requirements.txt with hashes. pip-sync installs exactly what’s in that file, removing anything extra.

This gives you a real lockfile without switching package managers. Great for teams not ready to adopt poetry or uv.

poetry

Dependency management + packaging + lockfile in one tool. poetry add httpx updates pyproject.toml, resolves deps, and updates poetry.lock. The lockfile stores all transitive deps with hashes.

Poetry’s resolver is good. Its build backend is solid. The pain points: slightly non-standard pyproject.toml format (predates PEP 621 full adoption), slower than uv, and the monolithic design means it’s harder to use just one piece of it.

pdm

PEP 621-compliant dependency manager with optional PEP 582 support (no virtualenv, packages in __pypackages__/). More modular than poetry. Good monorepo story. Smaller community than poetry.

uv

Rust-based, drop-in replacement for pip + pip-tools + virtualenv. 10–100x faster than pip in benchmarks. Handles virtual environment creation, package installation, and dependency resolution. First-class uv.lock lockfile format.

uv venv                          # create .venv
uv pip install httpx             # install package
uv pip compile requirements.in  # like pip-compile but faster
uv run pytest                    # run command inside venv without activating
uv sync                          # install from pyproject.toml + uv.lock
uv add httpx                     # add dep + update lockfile
uv build                         # build sdist + wheel
uv publish                       # publish to PyPI

uv is now the de-facto recommendation for new projects. It’s not opinionated about build backends — it works with any PEP 517-compliant backend.

Comparison

ELI5: pip is the old reliable but slowest checkout line at the grocery store. uv is the self-checkout that’s also 10x faster, doesn’t make you bag everything yourself, and keeps a perfect receipt for next time.

Dependency Resolution

Resolution is the process of finding a single set of package versions that satisfies all constraints from all your dependencies simultaneously. It’s a constraint satisfaction problem — at scale, it’s NP-hard.

When you run pip install "httpx>=0.27" "anyio>=4.0", pip has to find versions of httpx, anyio, and all their transitive dependencies that are mutually compatible.

Version Specifiers

In libraries: never pin exact versions. A library is installed alongside other packages — exact pins cause conflicts. Use >=minimum,<next-major (or just >=minimum).

In applications: pin everything. Your deployed app should install the same versions every time. That’s what lockfiles are for.

Common mistake: Using == in a library’s dependencies. Your library becomes nearly impossible to install alongside anything else. Use ~= or >=.

Lockfiles

A lockfile records the exact versions (and hashes) of every package installed, including transitives. If you commit the lockfile, everyone on the team gets bit-for-bit identical environments.

• poetry.lock — poetry’s format, JSON under the hood
• uv.lock — uv’s format, TOML, cross-platform
• requirements.txt (generated) — flat list of pinned packages, widely supported

Commit lockfiles for applications. Don’t commit them for libraries. Libraries should be tested across a range of dependency versions.

ELI5: A lockfile is like a photo of every ingredient in your dish the day it tasted perfect. Next time you cook, you don’t have to guess which version of “salt” was in the market — you buy exactly what’s in the photo.

Building & Distribution

Two artifacts you’ll publish to PyPI: source distributions and wheels.

sdist vs wheel

Source distribution (sdist): A .tar.gz of your source code. The installer has to build it locally, which means running your build backend, which means pip needs to have setuptools (or whatever) installed, and if you have C extensions, the user needs a C compiler.

Wheel: A pre-built, ready-to-install archive. Zip file with a .whl extension. No build step on the user’s machine. Always faster to install.

Wheel naming: {name}-{version}-{python_tag}-{abi_tag}-{platform_tag}.whl

Examples:

• httpx-0.27.0-py3-none-any.whl — pure Python, works everywhere
• numpy-2.0.0-cp311-cp311-linux_x86_64.whl — CPython 3.11, Linux x86_64 only

Publish both sdist and wheel. Build:

uv build           # or: python -m build
# produces: dist/my_package-1.0.0.tar.gz
#           dist/my_package-1.0.0-py3-none-any.whl

Publish:

uv publish                          # uses UV_PUBLISH_TOKEN env var
twine upload dist/*                 # classic approach

Private Registries

For internal packages, you don’t publish to PyPI. Common options:

• devpi — self-hosted PyPI mirror + registry
• AWS CodeArtifact — managed, integrates with IAM
• GCP Artifact Registry — managed, integrates with GCP auth
• GitHub Packages — free for public repos, easy CI integration

Configure pip/uv with --index-url or --extra-index-url to pull from private registries alongside PyPI.

Common mistake: Publishing an internal package to PyPI by accident. Always check your [build-system] and publishing config before running uv publish. Consider setting Private :: Do Not Upload trove classifier as a reminder.

Development Tooling

Linting & Formatting

ruff has won. It replaces flake8, isort, pyupgrade, pydocstyle, bandit (partially), and more — all in one Rust binary that runs 10–100x faster.

ruff check .           # lint
ruff check --fix .     # lint + autofix
ruff format .          # format (replaces black)

Configure in pyproject.toml:

[tool.ruff]
line-length = 88
target-version = "py311"

[tool.ruff.lint]
select = ["E", "F", "I", "UP", "B", "SIM"]
ignore = ["E501"]

[tool.ruff.format]
quote-style = "double"

ruff format is largely black-compatible. If you’re migrating from black, it’s nearly a drop-in.

mypy for strict type checking. pyright (from Microsoft) is faster and has better inference — VSCode uses it via pylance. For CI, either works.

ELI5: ruff is like replacing a toolbox with ten separate screwdrivers with one electric screwdriver that’s faster than all of them combined and fits in your pocket.

Pre-commit

pre-commit runs hooks before each commit. Install hooks once, they run automatically:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/astral-sh/ruff-pre-commit
    rev: v0.4.0
    hooks:
      - id: ruff
        args: [--fix]
      - id: ruff-format
  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.10.0
    hooks:
      - id: mypy

pre-commit install       # set up hooks
pre-commit run --all-files  # run manually

Task Runners

For new projects: just for simple tasks, nox if you need to test across Python 3.10/3.11/3.12.

Project Structure

src/ Layout vs Flat Layout

Flat layout:

my_package/
  __init__.py
  module.py
tests/
pyproject.toml

src/ layout:

src/
  my_package/
    __init__.py
    module.py
tests/
pyproject.toml

The key difference: with a flat layout, import my_package in tests can accidentally import your source directory directly instead of the installed package. This hides packaging bugs — your package might be missing files and you’d never know until a user installs it.

With src/ layout, my_package is only importable after installation (or pip install -e .), so your tests always test the installed artifact.

ELI5: Flat layout is like keeping your source code files and your compiled app in the same folder. You might accidentally run the wrong one. src/ layout keeps them separate — tests always use the installed version.

Common mistake: Choosing flat layout and then being confused why tests pass locally but fail after install because you forgot to add a data file to MANIFEST.in (sdist) or [tool.hatch.build] (hatchling).

Editable Installs

pip install -e . (or uv pip install -e .) installs your package in editable mode — changes to source files are immediately reflected without reinstalling. Modern editable installs use import hooks; older ones used .pth files.

Entry Points

[project.scripts]
my-cli = "my_package.cli:main"

[project.gui-scripts]
my-gui = "my_package.gui:launch"

When someone installs your package, my-cli becomes an executable in their bin/ that calls my_package.cli.main(). No shebangs, no if __name__ == "__main__" wrapper needed.

Plugins use [project.entry-points]:

[project.entry-points."pytest11"]
my-plugin = "my_package.pytest_plugin"

Summary — Decision Table