Metaprogramming

Metaprogramming is code that writes or modifies code. In Python that means: decorators, metaclasses, __init_subclass__, ABCs, dynamic attribute access, and code generation. Most engineers understand the syntax. Very few understand why the machinery works the way it does — and that gap causes subtle bugs.

Decorators — Beyond the Basics

Functions Are Objects

You already know this, but internalize it: a function is an object. You can store it in a variable, pass it to another function, return it from a function, put it in a list. A decorator is just syntactic sugar for passing a function as an argument.

def loud(func):
    def wrapper(*args, **kwargs):
        print(f"calling {func.__name__}")
        return func(*args, **kwargs)
    return wrapper

@loud
def greet(name): return f"hi {name}"

# Exactly equivalent to:
greet = loud(greet)

ELI5: A decorator is a gift-wrapper. You give it a present (the function), it wraps it in paper and a bow (extra behavior), and returns the wrapped gift. You get the wrapped version back, not the original. The wrapping happens at decoration time, not when the gift is opened (called).

Execution Order: Bottom-Up Application, Top-Down Execution

This trips everyone up the first time.

@A
@B
@C
def f(): ...
# Applied bottom-up: f = A(B(C(f)))
# Executed top-down when called: A's wrapper runs first, then B's, then C's

Applied bottom-up means C wraps f first, then B wraps that result, then A wraps that. At call time, A’s wrapper runs first (outermost), eventually calling B’s wrapper, which calls C’s wrapper, which calls f.

Common mistake: Expecting decorators to apply in declaration order (top to bottom). They don’t. Diagram it as a stack of onion layers.

Decorators with Arguments: The Triple-Nested Pattern

A decorator with arguments is a factory that returns a decorator.

def retry(times=3, exceptions=(Exception,)):
    def decorator(func):
        import functools
        @functools.wraps(func)
        def wrapper(*args, **kwargs):
            for attempt in range(times):
                try:
                    return func(*args, **kwargs)
                except exceptions as e:
                    if attempt == times - 1:
                        raise
                    print(f"retry {attempt+1}/{times}: {e}")
        return wrapper
    return decorator

@retry(times=5, exceptions=(IOError, TimeoutError))
def fetch(url): ...

The three layers: retry(...) returns decorator, which takes the function and returns wrapper. When you write @retry(times=5), Python first calls retry(times=5) to get the actual decorator, then applies it.

functools.wraps — Use It, Always

Without functools.wraps, your decorator replaces the wrapped function’s metadata:

def bare_decorator(func):
    def wrapper(*args, **kwargs):
        return func(*args, **kwargs)
    return wrapper

@bare_decorator
def important(): """Does important things."""

important.__name__   # 'wrapper'  — WRONG
important.__doc__    # None       — WRONG
important.__module__ # wherever bare_decorator is defined — WRONG

This breaks introspection, help(), logging, Sphinx, pytest output, and anything that reads __name__. Fix: @functools.wraps(func) on the wrapper. It copies __name__, __doc__, __module__, __qualname__, __annotations__, and __wrapped__.

ELI5: functools.wraps is like putting a name tag on the gift wrapper. Without it, the package just says “wrapper.” With it, the package still says “birthday present for Alice.”

Class Decorators

Two flavors — decorating a class, and decorating methods inside a class.

Decorating a class (gets the class object, returns a class):

def add_repr(cls):
    def __repr__(self):
        attrs = ', '.join(f'{k}={v!r}' for k, v in vars(self).items())
        return f'{cls.__name__}({attrs})'
    cls.__repr__ = __repr__
    return cls

@add_repr
class Point:
    def __init__(self, x, y):
        self.x, self.y = x, y

This is a lightweight alternative to metaclasses for modifying class behavior at definition time. dataclasses.dataclass works exactly this way.

Real-World Decorator Patterns

functools.singledispatch — Poor Man’s Overloading

from functools import singledispatch

@singledispatch
def process(arg):
    raise NotImplementedError(f"Cannot process {type(arg)}")

@process.register(int)
def _(arg): return arg * 2

@process.register(str)
def _(arg): return arg.upper()

@process.register(list)
def _(arg): return [process(x) for x in arg]

Dispatches on the type of the first argument. Works with inheritance — if no exact match, walks the MRO. Not as powerful as multimethods but covers 90% of real cases.

Metaclasses

Classes Are Instances of Their Metaclass

This is the fact that unlocks everything:

>>> type(int)
<class 'type'>
>>> type(str)
<class 'type'>
>>> type(type)
<class 'type'>

type is the metaclass of everything by default. A metaclass is the class of a class — it defines how classes behave at class creation time.

ELI5: A class is a factory that creates objects. A metaclass is a factory that creates factories. When Python sees class Foo:, it calls type('Foo', bases, namespace) to build the class object. A metaclass swaps out type for your custom factory.

type(name, bases, namespace) — Dynamic Class Creation

Dog = type('Dog', (object,), {
    'sound': 'woof',
    'speak': lambda self: self.sound,
})

d = Dog()
d.speak()  # 'woof'

This is identical to:

class Dog:
    sound = 'woof'
    def speak(self): return self.sound

Useful when you need to create classes programmatically from config, schemas, or code generation.

The Class Creation Sequence

When Python processes class Foo(Base, metaclass=Meta):, the sequence is:

1. Meta.__prepare__(name, bases) — returns the namespace dict (allows ordered or custom namespaces)
2. Execute class body — runs the class body code, populating the namespace
3. Meta.__new__(mcs, name, bases, namespace) — creates the class object
4. Meta.__init__(cls, name, bases, namespace) — initializes it

Most metaclasses only override __new__ or __init__. __prepare__ is needed when you want the class body to use a custom dict (e.g., ordered, or one that records definition order).

When You Actually Need Metaclasses

Honestly: rarely. The cases that genuinely require metaclasses:

Common mistake: Using a metaclass when a class decorator would do the job. Metaclass conflicts (two parents with different metaclasses) are painful to debug.

ORM-Like Field Registration: Two Approaches

# Metaclass approach
class ModelMeta(type):
    def __new__(mcs, name, bases, namespace):
        fields = {k: v for k, v in namespace.items()
                  if isinstance(v, Field)}
        cls = super().__new__(mcs, name, bases, namespace)
        cls._fields = fields
        return cls

class Model(metaclass=ModelMeta): pass

class User(Model):
    name = Field(str)
    age  = Field(int)

# __init_subclass__ approach (Python 3.6+, simpler)
class Model:
    def __init_subclass__(cls, **kwargs):
        super().__init_subclass__(**kwargs)
        cls._fields = {k: v for k, v in vars(cls).items()
                       if isinstance(v, Field)}

class User(Model):
    name = Field(str)
    age  = Field(int)

The __init_subclass__ version is shorter, has no metaclass, composes cleanly with other base classes, and handles 95% of ORM-style use cases.

__init_subclass__ — The Metaclass Killer

What It Does

__init_subclass__ is called on the parent class whenever a subclass is created. No metaclass needed.

class Plugin:
    _registry = {}

    def __init_subclass__(cls, plugin_name=None, **kwargs):
        super().__init_subclass__(**kwargs)
        name = plugin_name or cls.__name__.lower()
        Plugin._registry[name] = cls

class CSVPlugin(Plugin, plugin_name="csv"):
    def load(self, path): ...

class JSONPlugin(Plugin):  # registered as 'jsonplugin'
    def load(self, path): ...

Plugin._registry  # {'csv': CSVPlugin, 'jsonplugin': JSONPlugin}

ELI5: __init_subclass__ is a form you fill out automatically when your subclass is born. The parent class gets a copy and can record whatever it wants. No bureaucracy (metaclass) needed.

Why Prefer It Over Metaclasses

Always try __init_subclass__ first. Reach for a metaclass only when you need __prepare__ or need to intercept the namespace during class body execution.

Limitations

• Runs when the class statement executes (import time), not at instantiation
• Cannot modify the class namespace before the class body runs (that’s __prepare__)
• Does not intercept the creation of the base class itself — only subclasses

Abstract Base Classes

The Machinery

abc.ABCMeta is a metaclass that tracks which methods are abstract and prevents instantiation if any are unimplemented.

from abc import ABC, abstractmethod

class Shape(ABC):
    @abstractmethod
    def area(self) -> float: ...

    @abstractmethod
    def perimeter(self) -> float: ...

class Circle(Shape):
    def __init__(self, r): self.r = r
    def area(self): return 3.14159 * self.r ** 2
    def perimeter(self): return 2 * 3.14159 * self.r

Shape()   # TypeError: Can't instantiate abstract class Shape
Circle(5) # Fine — both abstract methods implemented

ABC is a convenience class that sets metaclass=ABCMeta. @abstractmethod sets func.__isabstractmethod__ = True.

Virtual Subclasses

register() tells an ABC “treat this class as a subclass even though it doesn’t inherit”:

class Drawable(ABC):
    @abstractmethod
    def draw(self): ...

class ThirdPartyWidget:  # doesn't inherit Drawable
    def draw(self): print("drawing widget")

Drawable.register(ThirdPartyWidget)
isinstance(ThirdPartyWidget(), Drawable)  # True
issubclass(ThirdPartyWidget, Drawable)    # True

__subclasshook__ is more dynamic — it lets you implement your own isinstance check:

class Drawable(ABC):
    @classmethod
    def __subclasshook__(cls, C):
        if cls is Drawable:
            return hasattr(C, 'draw')
        return NotImplemented

Now anything with a draw attribute is considered a Drawable. This is duck typing with isinstance semantics.

ABCs vs Protocols

Rule of thumb: In new code, use Protocol from typing for type-checker-enforced interfaces. Use ABC when you need isinstance checks at runtime, or when you want to provide mixin method implementations.

ELI5: ABC says “you must be a member of Club Shape, and you prove it by signing the membership form (inheriting).” Protocol says “you can come to the party if you can dance — I don’t care where you learned.” ABCs are for enforcing contracts. Protocols are for describing capabilities.

Common mistake: Using ABC for every interface. In Python 3.8+, Protocol is almost always the right choice for type annotations. ABCs add coupling; Protocols don’t.

Dynamic Attribute Access

__getattr__ vs __getattribute__

These two are confusingly named and do very different things.

__getattribute__ — called on every attribute access, before anything. Override this only if you need to intercept all attribute lookups, including existing ones. Dangerous: calling self.anything inside __getattribute__ recurses infinitely. Must use object.__getattribute__(self, name).

__getattr__ — called only when normal lookup fails (attribute not found in instance dict, class dict, or MRO). This is the right hook for most dynamic attribute work.

class Config:
    def __init__(self, data: dict):
        object.__setattr__(self, '_data', data)  # bypass __setattr__

    def __getattr__(self, name):
        try:
            return self._data[name]
        except KeyError:
            raise AttributeError(name)

    def __setattr__(self, name, value):
        self._data[name] = value

The attribute lookup order Python follows:

1. type(obj).__mro__   → data descriptors (has both __get__ and __set__)
2. obj.__dict__        → instance dict
3. type(obj).__mro__   → non-data descriptors and class attrs
4. __getattr__         → fallback (only if above all fail)

ELI5: __getattribute__ is the bouncer who checks every single person trying to enter — even your friends with VIP passes. __getattr__ is the “lost and found” window — only opens when nobody else can help you.

Building a Proxy Object

class ReadOnlyProxy:
    def __init__(self, target):
        object.__setattr__(self, '_target', target)

    def __getattr__(self, name):
        return getattr(self._target, name)

    def __setattr__(self, name, value):
        raise AttributeError("Read-only proxy")

Common mistake: Writing self._target = target in __init__ when you’ve overridden __setattr__. That calls your __setattr__, which raises. Use object.__setattr__(self, '_target', target) to bypass.

hasattr() Has Side Effects

hasattr(obj, name) is implemented as: try getattr(obj, name), return False if it raises AttributeError. This means:

• It calls __getattr__, which may have side effects
• It triggers descriptor protocol
• It can mask bugs: if __getattr__ raises AttributeError due to a bug inside (not because the attr is missing), hasattr returns False silently

Prefer explicit try/except in code that needs to be correct:

# Safer pattern
try:
    value = obj.some_attr
except AttributeError:
    value = default

exec(), eval(), and Code Generation

When It’s Appropriate

Code generation is appropriate when:

• Build-time only: generating Python from a schema/spec (SQLAlchemy does this for column accessors)
• Config-driven dispatch: building dispatch tables from YAML/JSON config
• namedtuple-style class factories: collections.namedtuple uses exec() internally to create a class with proper __repr__, _fields, etc.

Why It’s Almost Always Wrong in Production

# Never do this with user input
user_expr = request.get_json()['formula']
result = eval(user_expr)  # RCE waiting to happen

Problems with runtime exec/eval:

• Security: any Python code runs — including __import__('os').system('rm -rf /')
• Debugging: tracebacks point into generated code, not your source file
• Performance: compile → parse → eval on every call; can’t be optimized by the JIT
• Static analysis: mypy, pyright, IDEs can’t see inside the string

compile() and ast — The Safer Alternative

If you must generate and run code, compile it once and run many times:

code = compile("x * 2 + y", "<expr>", "eval")
result = eval(code, {}, {"x": 10, "y": 5})  # 25

For safe expression evaluation (math, filter expressions), parse to AST and evaluate only safe node types:

import ast

def safe_eval(expr: str, variables: dict):
    tree = ast.parse(expr, mode='eval')
    # Walk tree, raise if any node is Call, Attribute, Import, etc.
    for node in ast.walk(tree):
        if not isinstance(node, (ast.Expression, ast.BinOp, ast.UnaryOp,
                                  ast.Constant, ast.Name, ast.Load,
                                  ast.Add, ast.Sub, ast.Mult, ast.Div)):
            raise ValueError(f"Unsafe node: {type(node).__name__}")
    return eval(compile(tree, '<expr>', 'eval'), {}, variables)

ELI5: eval on user input is handing a stranger the keys to your house and saying “just get what you need.” The ast approach is searching their bag at the door and only letting through what you know is safe.

Common mistake: Using eval for simple config expressions when ast.literal_eval (safe, evaluates only literals) or a proper expression library would suffice.

__class_getitem__ and Parameterized Types

How list[int] Works

Before Python 3.9, list[int] raised TypeError. In 3.9+, it works because list defines __class_getitem__:

>>> list.__class_getitem__(int)
list[int]

__class_getitem__ is a class method called when you do ClassName[item]. It returns a types.GenericAlias (or your own object).

Building Your Own Parameterized Class

class TypedList:
    def __class_getitem__(cls, item):
        # item is whatever was in the brackets
        return type(f'TypedList[{item.__name__}]', (cls,), {'_type': item})

    def __init__(self):
        self._data = []

    def append(self, value):
        if not isinstance(value, self._type):
            raise TypeError(f"Expected {self._type}, got {type(value)}")
        self._data.append(value)

IntList = TypedList[int]
lst = IntList()
lst.append(1)   # fine
lst.append("x") # TypeError

This is how typing.Generic works under the hood — __class_getitem__ creates a parameterized version of the class that carries type information. At runtime it’s mostly for documentation and type checkers; the actual type enforcement is left to you or the type checker.

Metaprogramming Decision Framework

When you hit a problem that feels like it needs metaprogramming, walk this ladder:

Problem: "I need to modify behavior at class/function definition time"

Can a plain function or class method solve it?
  YES → Use that

Can a decorator (function or class) solve it?
  YES → Use a decorator

Does the behavior need to be inherited by subclasses?
  YES → Can __init_subclass__ handle it?
    YES → Use __init_subclass__
    NO  → Do you need to intercept the class namespace during class body execution?
      YES → Use a metaclass
      NO  → You probably still want __init_subclass__

Is the code fully static (build-time, not hot path)?
  YES → Code generation via ast / exec is acceptable
  NO  → Find a data-driven solution instead

Summary Table

The real rule: every layer up this stack doubles the debugging time and halves the number of engineers who can maintain it. Use the lowest layer that solves your problem. Metaclasses that could be __init_subclass__ are a code smell. exec that could be a dict lookup is a security hole.