Notes
Longer pieces, roughly newest first. No schedule.
Percentiles lie about tails
A p95 is a summary of the part of the distribution you were already watching. When the interesting failure lives past p99, tracking p95 tells you almost nothing — and it will look reassuringly stable while it happens.
Sampling a signal you cannot pause
Every probe costs something. If the probe itself perturbs the system, interval choice stops being a statistics question and becomes a budget question.
Two points beat one
A single timing includes fixed overhead you cannot see. Two measurements of different sizes let you subtract it out: the slope is the answer, the intercept is the noise you were about to publish.
Averages of ratios
Averaging per-request rates and computing a rate from totals give different
numbers. One of them is usually the one you meant. Worked example with
sum(x)/sum(t) versus mean(x/t).