Python Packaging

Python packaging enables distribution of code as reusable libraries and applications. This skill covers modern packaging standards using pyproject.toml, Poetry for dependency management, setuptools for build configuration, semantic versioning, and publishing to PyPI. Understanding packaging is essential for creating maintainable libraries, sharing code across projects, and managing dependencies in production applications.

Key Concepts

• pyproject.toml (PEP 518) - Modern configuration file for Python projects replacing setup.py and setup.cfg
• Poetry - Dependency management and packaging tool with lock files and virtual environment management
• setuptools - Build backend for creating distributions (sdist, wheel) from Python packages
• Semantic Versioning - Version numbering scheme (MAJOR.MINOR.PATCH) for backward compatibility signaling
• Source Distribution (sdist) - Archive containing source code and metadata for building packages
• Wheel Distribution - Built binary distribution for faster installation without compilation
• Package Index (PyPI) - Public repository for Python packages; private indexes for internal distribution
• Entry Points - Declared console scripts and plugin discovery mechanisms
• Dependency Specification - Version constraints (^, ~, >=) for reproducible installations
• Lock Files - Poetry.lock or requirements.txt pins exact versions for deterministic builds

Best Practices

1. Use pyproject.toml - Adopt modern PEP 518 configuration over legacy setup.py files
2. Manage dependencies with Poetry - Use Poetry for consistent dependency resolution and lock files
3. Follow semantic versioning - Increment MAJOR for breaking changes, MINOR for features, PATCH for fixes
4. Pin dependencies in production - Use lock files (Poetry.lock) to ensure reproducible deployments
5. Specify Python version constraints - Declare minimum Python version and test against supported versions
6. Include metadata - Provide comprehensive project metadata (description, authors, license, URLs)
7. Separate dev dependencies - Keep development tools separate from production dependencies
8. Build both sdist and wheel - Distribute both formats for maximum compatibility and performance
9. Test before publishing - Use TestPyPI for pre-release validation before publishing to PyPI
10. Automate releases - Use CI/CD pipelines for version bumps, building, and publishing

Code Examples

Modern pyproject.toml Configuration

# ✅ GOOD: Comprehensive pyproject.toml with Poetry
[tool.poetry]
name = "mypackage"
version = "1.2.3"
description = "A production-grade Python package"
authors = ["Your Name <you@example.com>"]
license = "MIT"
readme = "README.md"
homepage = "https://github.com/username/mypackage"
repository = "https://github.com/username/mypackage"
documentation = "https://mypackage.readthedocs.io"
keywords = ["api", "client", "sdk"]
classifiers = [
    "Development Status :: 5 - Production/Stable",
    "Intended Audience :: Developers",
    "License :: OSI Approved :: MIT License",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.11",
    "Programming Language :: Python :: 3.12",
    "Topic :: Software Development :: Libraries",
]

# Core dependencies
[tool.poetry.dependencies]
python = "^3.11"  # Minimum Python 3.11
httpx = "^0.26.0"  # Compatible with 0.26.x
pydantic = "^2.5.0"
python-dateutil = "^2.8.0"

# Optional dependencies for features
[tool.poetry.extras]
async = ["aiohttp", "aiodns"]
orjson = ["orjson"]
all = ["aiohttp", "aiodns", "orjson"]

# Development dependencies
[tool.poetry.group.dev.dependencies]
pytest = "^7.4.0"
pytest-cov = "^4.1.0"
mypy = "^1.8.0"
ruff = "^0.1.0"
black = "^24.0.0"

# Documentation dependencies
[tool.poetry.group.docs.dependencies]
sphinx = "^7.2.0"
sphinx-rtd-theme = "^2.0.0"

# Console scripts
[tool.poetry.scripts]
mypackage = "mypackage.cli:main"
mypackage-admin = "mypackage.admin:cli"

# Build system
[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

# Black configuration
[tool.black]
line-length = 88
target-version = ['py311']
include = '\.pyi?$'

# mypy configuration
[tool.mypy]
python_version = "3.11"
strict = true
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true

# pytest configuration
[tool.pytest.ini_options]
testpaths = ["tests"]
python_files = ["test_*.py"]
python_functions = ["test_*"]
addopts = "--cov=mypackage --cov-report=term-missing"

# Coverage configuration
[tool.coverage.run]
source = ["mypackage"]
branch = true

[tool.coverage.report]
exclude_lines = [
    "pragma: no cover",
    "if TYPE_CHECKING:",
    "raise NotImplementedError",
]

# ❌ BAD: Legacy setup.py with incomplete metadata
"""
from setuptools import setup

setup(
    name="mypackage",
    version="1.0.0",
    packages=["mypackage"],
    install_requires=[
        "requests",  # No version constraint
    ]
)
"""

Package Structure and init.py

# ✅ GOOD: Proper package structure with version and exports
"""
mypackage/
├── pyproject.toml
├── README.md
├── LICENSE
├── src/
│   └── mypackage/
│       ├── __init__.py
│       ├── __version__.py
│       ├── client.py
│       ├── models.py
│       ├── exceptions.py
│       └── py.typed  # PEP 561 type hints marker
├── tests/
│   ├── __init__.py
│   ├── test_client.py
│   └── test_models.py
└── docs/
    └── index.md
"""

# src/mypackage/__version__.py
"""Version information for mypackage."""

__version__ = "1.2.3"
__version_info__ = tuple(int(i) for i in __version__.split("."))

# src/mypackage/__init__.py
"""
MyPackage - A production-grade Python package.

Example:
    >>> from mypackage import Client
    >>> client = Client(api_key="secret")
    >>> result = client.fetch_data()
"""

from mypackage.__version__ import __version__, __version_info__
from mypackage.client import Client, AsyncClient
from mypackage.exceptions import (
    MyPackageError,
    APIError,
    ValidationError,
)
from mypackage.models import (
    User,
    Organization,
    Resource,
)

__all__ = [
    # Version
    "__version__",
    "__version_info__",
    # Clients
    "Client",
    "AsyncClient",
    # Exceptions
    "MyPackageError",
    "APIError",
    "ValidationError",
    # Models
    "User",
    "Organization",
    "Resource",
]

# ❌ BAD: Flat structure with empty __init__.py
"""
mypackage/
├── setup.py
├── client.py
├── models.py
└── __init__.py  # Empty file
"""

Dependency Management with Poetry

# ✅ GOOD: Poetry workflow for dependency management

# Initialize new project
poetry new mypackage
cd mypackage

# Or initialize in existing project
poetry init

# Add production dependency
poetry add httpx
poetry add "pydantic>=2.5,<3.0"

# Add development dependency
poetry add --group dev pytest pytest-cov mypy

# Add optional dependency
poetry add --optional aiohttp
poetry add --optional orjson

# Update dependencies
poetry update  # Update all to latest compatible versions
poetry update httpx  # Update specific package

# Show dependency tree
poetry show --tree

# Export requirements.txt for legacy tools
poetry export -f requirements.txt --output requirements.txt --without-hashes

# Install project in editable mode
poetry install

# Install only production dependencies
poetry install --only main

# Build distribution packages
poetry build  # Creates sdist and wheel in dist/

# Publish to PyPI
poetry publish --build

# Publish to TestPyPI first
poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry publish -r testpypi --build

# ❌ BAD: Manual pip with unpinned versions
"""
pip install requests
pip install pytest
# No lock file, no reproducibility
"""

Semantic Versioning Strategy

# ✅ GOOD: Clear versioning with changelog
"""
# CHANGELOG.md

## [2.0.0] - 2026-02-01
### Breaking Changes
- Remove deprecated `Client.old_method()` (use `new_method()` instead)
- Change return type of `fetch_data()` from dict to DataModel

### Added
- New `AsyncClient` for async/await support
- Support for Python 3.12

### Fixed
- Fix memory leak in connection pool

## [1.5.0] - 2026-01-15
### Added
- Add retry mechanism with exponential backoff
- Add timeout configuration

### Deprecated
- `Client.old_method()` is deprecated, will be removed in 2.0.0

## [1.4.1] - 2026-01-10
### Fixed
- Fix type hints for Python 3.11
- Fix connection timeout handling

## [1.4.0] - 2026-01-05
### Added
- Add support for pagination
- Add request logging
"""

# Version bumping strategy
"""
# Patch release (1.4.0 → 1.4.1)
# - Bug fixes
# - Security patches
# - Documentation updates
poetry version patch

# Minor release (1.4.1 → 1.5.0)
# - New features
# - Deprecations (with warnings)
# - Backward compatible changes
poetry version minor

# Major release (1.5.0 → 2.0.0)
# - Breaking API changes
# - Remove deprecated features
# - Architecture changes
poetry version major

# Pre-release versions
poetry version prerelease  # 2.0.0 → 2.0.0-alpha.0
poetry version patch       # 2.0.0-alpha.0 → 2.0.0-alpha.1
"""

# ❌ BAD: Random version numbers without strategy
"""
1.0.0 → 1.0.1: Add new feature (should be 1.1.0)
1.0.1 → 2.0.0: Fix typo in docs (should be 1.0.2)
2.0.0 → 2.0.1: Breaking API change (should be 3.0.0)
"""

Entry Points and Console Scripts

# ✅ GOOD: Defined console scripts in pyproject.toml
"""
[tool.poetry.scripts]
mypackage = "mypackage.cli:main"
mypackage-admin = "mypackage.admin:cli"
"""

# src/mypackage/cli.py
"""Command-line interface for mypackage."""

import argparse
import sys
from typing import Sequence

from mypackage import __version__
from mypackage.client import Client

def main(argv: Sequence[str] | None = None) -> int:
    """
    Main entry point for CLI.
    
    Args:
        argv: Command-line arguments (defaults to sys.argv)
        
    Returns:
        Exit code (0 for success, non-zero for errors)
    """
    parser = argparse.ArgumentParser(
        prog="mypackage",
        description="MyPackage CLI tool"
    )
    parser.add_argument(
        "--version",
        action="version",
        version=f"%(prog)s {__version__}"
    )
    parser.add_argument(
        "--api-key",
        required=True,
        help="API key for authentication"
    )
    parser.add_argument(
        "command",
        choices=["fetch", "list", "delete"],
        help="Command to execute"
    )
    
    args = parser.parse_args(argv)
    
    try:
        client = Client(api_key=args.api_key)
        
        if args.command == "fetch":
            data = client.fetch_data()
            print(f"Fetched {len(data)} items")
        elif args.command == "list":
            items = client.list_items()
            for item in items:
                print(f"- {item}")
        elif args.command == "delete":
            client.delete_all()
            print("Deleted all items")
        
        return 0
    
    except Exception as e:
        print(f"Error: {e}", file=sys.stderr)
        return 1

if __name__ == "__main__":
    sys.exit(main())

# After installation with poetry install:
# $ mypackage --api-key secret fetch
# Fetched 42 items

# ❌ BAD: No entry points, requires python -m
"""
python -m mypackage.cli --api-key secret fetch
"""

Publishing to PyPI

# ✅ GOOD: Automated publishing workflow
"""
# .github/workflows/publish.yml
name: Publish to PyPI

on:
  release:
    types: [published]

jobs:
  publish:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      
      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      
      - name: Install Poetry
        run: |
          curl -sSL https://install.python-poetry.org | python3 -
          echo "$HOME/.local/bin" >> $GITHUB_PATH
      
      - name: Verify version matches tag
        run: |
          VERSION=$(poetry version -s)
          TAG=${GITHUB_REF#refs/tags/v}
          if [ "$VERSION" != "$TAG" ]; then
            echo "Version mismatch: $VERSION != $TAG"
            exit 1
          fi
      
      - name: Build package
        run: poetry build
      
      - name: Test with TestPyPI
        run: |
          poetry config repositories.testpypi https://test.pypi.org/legacy/
          poetry publish -r testpypi --username __token__ --password ${{ secrets.TEST_PYPI_TOKEN }}
      
      - name: Publish to PyPI
        run: |
          poetry publish --username __token__ --password ${{ secrets.PYPI_TOKEN }}
"""

# Manual publishing steps
"""
# 1. Update version
poetry version minor

# 2. Update CHANGELOG.md with changes

# 3. Commit changes
git add pyproject.toml CHANGELOG.md
git commit -m "Bump version to $(poetry version -s)"

# 4. Create git tag
git tag -a "v$(poetry version -s)" -m "Release $(poetry version -s)"
git push origin main --tags

# 5. Build distributions
poetry build
ls dist/
# mypackage-1.2.3.tar.gz
# mypackage-1.2.3-py3-none-any.whl

# 6. Test with TestPyPI
poetry config repositories.testpypi https://test.pypi.org/legacy/
poetry publish -r testpypi

# 7. Install from TestPyPI and test
pip install --index-url https://test.pypi.org/simple/ mypackage==1.2.3

# 8. Publish to PyPI
poetry publish

# 9. Verify installation
pip install mypackage==1.2.3
"""

# ❌ BAD: Manual upload without verification
"""
python setup.py sdist
twine upload dist/*
# No testing, no version verification
"""

Type Hints Distribution (PEP 561)

# ✅ GOOD: Include type hints in package
"""
src/mypackage/
├── __init__.py
├── py.typed  # Marker file for PEP 561
├── client.py
└── models.py
"""

# pyproject.toml
"""
[tool.poetry]
packages = [
    { include = "mypackage", from = "src" }
]
include = [
    { path = "src/mypackage/py.typed", format = "sdist" },
    { path = "src/mypackage/py.typed", format = "wheel" }
]
"""

# src/mypackage/client.py with comprehensive types
from typing import TypeAlias, Protocol
from collections.abc import Sequence

UserId: TypeAlias = int

class Client:
    """Type-hinted client for external API."""
    
    def __init__(self, api_key: str, *, timeout: int = 30) -> None:
        self.api_key: str = api_key
        self.timeout: int = timeout
    
    def fetch_users(self, ids: Sequence[UserId]) -> list[dict[str, str | int]]:
        """Fetch users by IDs."""
        ...

# Users of the package get type checking
"""
from mypackage import Client

client: Client = Client(api_key="secret")
users: list[dict[str, str | int]] = client.fetch_users([1, 2, 3])
# mypy validates types without stub files
"""

# ❌ BAD: No type hints, requires external stubs
"""
def fetch_users(ids):
    # No types, users need to create .pyi stubs
    ...
"""

Anti-Patterns

Unpinned Dependencies in Production

# ❌ Avoid: No version constraints
[tool.poetry.dependencies]
requests = "*"  # Any version - dangerous!
pydantic = ">=1.0"  # Too broad range

# ✅ Fix: Use semantic version constraints
[tool.poetry.dependencies]
requests = "^2.31.0"  # Compatible with 2.x, >= 2.31.0
pydantic = "^2.5.0"  # Compatible with 2.x, >= 2.5.0

Missing Metadata

# ❌ Avoid: Incomplete package metadata
[tool.poetry]
name = "mypackage"
version = "1.0.0"

# ✅ Fix: Comprehensive metadata
[tool.poetry]
name = "mypackage"
version = "1.0.0"
description = "A clear, concise description"
authors = ["Name <email@example.com>"]
license = "MIT"
readme = "README.md"
homepage = "https://github.com/user/repo"
classifiers = [...]

Including Test Files in Distribution

# ❌ Avoid: Including unnecessary files
[tool.poetry]
packages = [
    { include = "*" }  # Includes everything!
]

# ✅ Fix: Explicitly include only package
[tool.poetry]
packages = [
    { include = "mypackage", from = "src" }
]
exclude = ["tests", "docs", "*.pyc"]

No Version Bumping Strategy

# ❌ Avoid: Manual version editing
# Edit pyproject.toml manually and typo version

# ✅ Fix: Use poetry version command
poetry version patch  # 1.0.0 → 1.0.1
poetry version minor  # 1.0.1 → 1.1.0
poetry version major  # 1.1.0 → 2.0.0

Testing Strategies

Testing Package Installation

# tests/test_package.py
import subprocess
import sys
from pathlib import Path

def test_package_builds_successfully() -> None:
    """Test that package builds without errors."""
    result = subprocess.run(
        ["poetry", "build"],
        capture_output=True,
        text=True
    )
    assert result.returncode == 0
    assert Path("dist").exists()
    
    # Verify both sdist and wheel created
    dist_files = list(Path("dist").glob("*"))
    assert any(f.suffix == ".gz" for f in dist_files)  # sdist
    assert any(f.suffix == ".whl" for f in dist_files)  # wheel

def test_package_installs_in_clean_environment() -> None:
    """Test installation in isolated environment."""
    # Build package
    subprocess.run(["poetry", "build"], check=True)
    
    # Create virtual environment
    venv_path = Path("test_venv")
    subprocess.run([sys.executable, "-m", "venv", str(venv_path)], check=True)
    
    # Install built package
    wheel = next(Path("dist").glob("*.whl"))
    pip = venv_path / "bin" / "pip"
    subprocess.run([str(pip), "install", str(wheel)], check=True)
    
    # Verify import works
    python = venv_path / "bin" / "python"
    result = subprocess.run(
        [str(python), "-c", "import mypackage; print(mypackage.__version__)"],
        capture_output=True,
        text=True,
        check=True
    )
    assert result.stdout.strip() != ""

def test_console_scripts_installed() -> None:
    """Test that console scripts are accessible."""
    result = subprocess.run(
        ["mypackage", "--version"],
        capture_output=True,
        text=True
    )
    assert result.returncode == 0
    assert "mypackage" in result.stdout

Testing with Multiple Python Versions

# ✅ GOOD: Test matrix in CI
"""
# .github/workflows/test.yml
strategy:
  matrix:
    python-version: ['3.11', '3.12']
    os: [ubuntu-latest, macos-latest, windows-latest]
"""

# Using tox for local multi-version testing
"""
# tox.ini
[tox]
envlist = py311,py312

[testenv]
deps =
    pytest
    pytest-cov
commands =
    pytest tests/ --cov=mypackage

# Run tests across all environments
tox
"""

Validating Package Metadata

# tests/test_metadata.py
import importlib.metadata as metadata

def test_package_metadata() -> None:
    """Verify package metadata is complete."""
    pkg_metadata = metadata.metadata("mypackage")
    
    assert pkg_metadata["Name"] == "mypackage"
    assert pkg_metadata["Version"]
    assert pkg_metadata["Author"]
    assert pkg_metadata["License"]
    assert pkg_metadata["Home-page"]

def test_version_consistency() -> None:
    """Verify version matches across files."""
    import mypackage
    
    pkg_version = metadata.version("mypackage")
    code_version = mypackage.__version__
    
    assert pkg_version == code_version

References

• Python Packaging User Guide
• PEP 518 - pyproject.toml
• Poetry Documentation
• Semantic Versioning
• PEP 561 - Distributing Type Information
• PyPI Publishing Guide
• TestPyPI
• setuptools Documentation

Related Skills

• python-best-practices.md - Project structure and configuration
• python-testing.md - Testing packages before publication
• async-python.md - Packaging async libraries
• pythonic-patterns.md - API design for public packages