Benchmark reproducibly¶

Now that the word counter has a correctness oracle, we can ask how long it takes. A useful benchmark is an experiment: it fixes the useful work, chooses an appropriate clock, repeats observations, and records enough context to explain variation.

Objectives

Distinguish elapsed timing, microbenchmarking, and interactive timing.
Construct a repeated benchmark around a representative workload.
Interpret a distribution without treating one measurement as universal.

Instructor note

15 min teaching
15 min type-along
5 min exercise

Choose the boundary before the clock¶

Decide whether the measurement includes input, setup, compilation, communication, or only a kernel. time.perf_counter() is a high-resolution elapsed-time clock suitable for an end-to-end region:

from pathlib import Path
import sys
from time import perf_counter

code_dir = next(
    path.resolve()
    for path in (Path("content/episodes/code"), Path("../content/episodes/code"))
    if path.is_dir()
)
if str(code_dir) not in sys.path:
    sys.path.insert(0, str(code_dir))

from wordcount import count_words, make_workload

text = make_workload(2_000)
start = perf_counter()
result = count_words(text)
elapsed = perf_counter() - start
print(f"{elapsed:.6f} s; {sum(result.values())} words")

Keep a use of result inside the experiment so the useful work is explicit. Do not compare this one observation with a number from another machine.

Run the cell a few times without changing it. The workload and answer stay the same while the elapsed time moves. A timer is working correctly when it reports what happened during each run; it is not expected to manufacture an identical result. Our job is to design enough repetition to describe that variation.

Type-along: repeat a microbenchmark¶

timeit reduces common timing mistakes and repeats a callable in a controlled loop. Inspect and run benchmark_wordcount.py:

$ python benchmark_wordcount.py

The script prints every seconds-per-call sample followed by the median. Setup happens outside the timed callable, both implementations receive the same immutable text, and each sample performs the same work. Successive samples reverse which implementation runs first to reduce systematic order bias. Retain those observations for the performance note in the final episode.

The same benchmark function can be explored directly in a notebook with a shorter workload. First predict whether the two sample lists will be identical and whether the first observation must be the slowest:

from benchmark_wordcount import benchmark
from statistics import median

samples = benchmark(repeats=5, number=2, workload_repeats=1_000)
for name, observations in samples.items():
    print(name, "raw:", [f"{value:.6f}" for value in observations])
    print(name, "median:", f"{median(observations):.6f} s/call")

Neither pattern is guaranteed. The raw values matter because a median alone can hide an outlier, a warm-up effect, or a noisy allocation.

In IPython or Jupyter, the equivalent tools are:

%time result = count_words(text)
%timeit count_words(text)

%time performs one run and is useful for long tasks; %timeit selects loops and repeats for shorter expressions. Terminal commands do not understand % magics.

Variation is information¶

Frequency scaling, other processes, memory placement, filesystem caches, runtime warm-up, and shared-node contention all affect measurements. Record at least:

code revision and dependency environment;
input and workload size;
machine, allocation, Python version, and thread settings;
metric, timing boundary, warm-up, repetitions, and summary statistic;
correctness check and the raw samples when possible.

For very short functions, time many calls per sample. For long applications, use fewer repetitions and retain individual observations. Avoid changing the workload merely to produce a dramatic speedup.

Make comparisons symmetric¶

A fair comparison keeps everything except the implementation under study as similar as possible. Use the same input object, correctness criterion, process placement, thread settings, repetition policy, and timing boundary. Alternate the order or repeat the whole experiment if thermal or allocation drift may favour whichever implementation runs first. When input preparation is part of the real application cost, measure it in a separate end-to-end experiment rather than quietly adding it to only one candidate.

Exercise

Run benchmark_wordcount.py twice. Are the printed values identical? Write one environmental or experimental reason for the difference and one fact you would include in a report.

Solution

Exact values normally differ. Scheduler activity, CPU frequency, caches, and competing work are possible reasons. A useful report might include the processor/node, fixed workload size, Python environment, number of repetitions, and all sample values or their median and spread.

Keypoints

Fix useful work and correctness before choosing a timing method.
Repetition describes variability; it does not make unrelated environments comparable.
Report the workload, environment, boundary, samples, and statistic with every claim.