Debugging and Profiling

Prerequisite: Writing Clean, Testable Code

Every programmer spends a large fraction of their time not writing code, but figuring out why existing code does not work. Debugging is not guesswork - it is a methodical, scientific process. Profiling is the same mindset applied to performance: measure first, then fix. This post covers the tools and habits that make both feel systematic rather than stressful.

The Debugging Mindset

Good debugging follows the scientific method:

1. Observe the unexpected behaviour.
2. Form a hypothesis about what might cause it.
3. Design a test that would confirm or refute the hypothesis.
4. Run the test, observe the result.
5. Repeat.

The key discipline is not jumping to a fix before you understand the cause. A guess that happens to work leaves you not knowing why, which means the same bug - or a variant - comes back.

Reading Tracebacks

When Python raises an exception, read the traceback from the bottom up. The last line names the exception and its message. The lines above show the call chain that led there - the frame at the top of that list is where your code was when things went wrong.

Traceback (most recent call last):
  File "main.py", line 10, in <module>
    result = process(data)
  File "main.py", line 5, in process
    return data["key"] * 2
KeyError: 'key'

Start at the bottom: KeyError: 'key'. Then look at the frame just above: data["key"] in process. That is where to look first.

Common Errors and What They Mean

• NameError: you referenced a variable name that does not exist in scope. Often a typo or a variable defined inside a block that you tried to use outside it.
• TypeError: an operation received the wrong type. "3" + 3 raises TypeError because you cannot add a string and an integer.
• AttributeError: you called a method or accessed an attribute that does not exist on that object. None.split() raises AttributeError. Usually means a function returned None when you expected an object.
• IndexError: you accessed a list at an index that does not exist. arr[5] on a three-element list.

The Python Debugger: pdb

print() debugging works, but a proper debugger is faster. Python ships with pdb. The easiest entry point is:

def process(items):
    total = 0
    for item in items:
        breakpoint()   # execution pauses here; Python 3.7+
        total += item
    return total

When execution hits breakpoint(), you drop into the debugger. Key commands:

Most IDEs (VS Code, PyCharm) wrap pdb in a graphical interface - you click to set breakpoints and inspect variables in a sidebar. Under the hood it is the same mechanism.

Assertions

Use assert to document and enforce invariants - conditions that should always be true:

def compute_average(numbers: list[float]) -> float:
    assert len(numbers) > 0, "Cannot average an empty list"
    return sum(numbers) / len(numbers)

When the condition is False, Python raises AssertionError with your message. Assertions are a lightweight way to catch bugs close to their source. They can be disabled globally with python -O, so do not use them to validate user input - use explicit if checks and raise ValueError or TypeError for that.

Logging

print() statements are temporary. The logging module is structured, configurable, and stays in your codebase without noise:

import logging

logging.basicConfig(
    level=logging.DEBUG,
    format="%(asctime)s %(levelname)s %(message)s"
)

logger = logging.getLogger(__name__)

def process(data):
    logger.debug("process called with %d items", len(data))
    result = [x * 2 for x in data]
    logger.info("processing complete")
    return result

Log levels, in increasing severity: DEBUG, INFO, WARNING, ERROR, CRITICAL. In production you typically show INFO and above; during development, DEBUG. You can route different loggers to different files, filter by module, or turn off logging entirely - none of which is possible with print().

Profiling: Find the Bottleneck First

The 80/20 rule applies to performance: 80% of execution time is spent in 20% of the code. Optimise that 20% - guessing where the bottleneck is usually wrong. Always measure first.

cProfile

cProfile is Python’s built-in profiler. It instruments every function call and reports how much time each one consumed:

import cProfile
import pstats

def slow_function(n):
    return sum(i * i for i in range(n))

with cProfile.Profile() as pr:
    slow_function(1_000_000)

stats = pstats.Stats(pr)
stats.sort_stats("cumulative")
stats.print_stats(10)   # show top 10 functions by cumulative time

Or run it directly from the command line without modifying your code:

python -m cProfile -s cumtime my_script.py

line_profiler and memory_profiler

cProfile shows which function is slow. line_profiler shows which line inside that function. Install with pip install line-profiler, then decorate the function you want to examine with @profile and run with kernprof -l -v script.py.

memory_profiler does the same for memory usage: pip install memory-profiler, @profile, run with python -m memory_profiler script.py.

Common Python Pitfalls

Mutable default arguments: Python evaluates default arguments once at function definition time, not each call.

# Bug: all calls share the same list object
def append_item(item, result=[]):
    result.append(item)
    return result

# Fix: use None as the sentinel
def append_item(item, result=None):
    if result is None:
        result = []
    result.append(item)
    return result

Reference vs copy: assigning a list does not copy it.

a = [1, 2, 3]
b = a        # b points to the same list
b.append(4)
print(a)     # [1, 2, 3, 4] - a was also modified!

b = a[:]     # shallow copy
b = a.copy() # also shallow copy

Examples

Debug a Buggy Binary Search

Here is a binary search with an off-by-one error. Step through it with pdb to find the bug.

def binary_search(arr: list[int], target: int) -> int:
    lo, hi = 0, len(arr)   # Bug: should be len(arr) - 1

    while lo <= hi:
        mid = (lo + hi) // 2
        if arr[mid] == target:
            return mid
        elif arr[mid] < target:
            lo = mid + 1
        else:
            hi = mid - 1

    return -1

Set breakpoint() before the loop, then use n and p lo, hi, mid on each iteration to watch the indices until you see arr[mid] raise an IndexError when hi equals len(arr) and mid goes out of bounds.

Replace Print-Debugging with Logging

import logging
logger = logging.getLogger(__name__)

def compute(values: list[float]) -> float:
    logger.debug("Input values: %s", values)
    filtered = [v for v in values if v > 0]
    logger.debug("After filtering: %d items remain", len(filtered))

    if not filtered:
        logger.warning("No positive values found, returning 0")
        return 0.0

    result = sum(filtered) / len(filtered)
    logger.info("Result: %.4f", result)
    return result

In production, set the log level to INFO. When debugging a reported issue, switch to DEBUG without changing any application logic.

Profile a Slow Function

import cProfile
import pstats

def find_duplicates(items):
    duplicates = []
    for i, item in enumerate(items):
        for j in range(i + 1, len(items)):  # O(n^2)
            if items[i] == items[j] and item not in duplicates:
                duplicates.append(item)
    return duplicates

data = list(range(1000)) + list(range(500))

with cProfile.Profile() as pr:
    find_duplicates(data)

pstats.Stats(pr).sort_stats("cumulative").print_stats(5)

The profile will show that find_duplicates dominates, pointing you to rewrite it with a set for $O(n)$ performance.

Debugging and profiling are skills built through practice. The tools are simple; the discipline - forming hypotheses, measuring before optimising, logging instead of printing - is what makes the difference.

Read Next: Operating Systems