Measure the pipeline before you optimise it

The instinct when a pipeline takes forty minutes is to start fixing it. Someone has a theory — the tests are slow, the image build is bloated, the runners are underpowered — and a week later the pipeline takes thirty-eight minutes and nobody is sure which change helped.

The problem is that the theory came before the measurement. Pipelines are one of the few systems where the instrumentation is free and nobody uses it.

Measure stages, across many runs

Two things matter and both get skipped.

Per stage, not total. A forty-minute number tells you nothing actionable. You need checkout, dependency resolution, test, image build, push and deploy as separate figures, because the fix for each is completely different.

Across thirty runs, not one. This is the part people miss. A single run tells you what happened once. The distribution tells you which problem you actually have. If the median is twelve minutes and the p90 is forty, you do not have a slow pipeline — you have a flaky one, and optimising stage duration will not touch it. If median and p90 sit close together, the pipeline is consistently slow and stage work will pay off.

Those are opposite problems with opposite fixes, and the average conceals which one you are looking at.

Most CI systems expose this without extra tooling. Jenkins has per-stage timings in the pipeline view. GitHub Actions reports step durations in the run summary and through its API. CircleCI exposes it in Insights. Pull thirty runs, put the stages in a spreadsheet, and the answer is usually obvious within an hour.

The four places the time hides

Once you have the numbers, findings cluster:

Dependency resolution with a cache that never hits. By far the most common. The cache is configured, everyone assumes it works, and the hit rate is near zero because the key includes something that changes every run — a timestamp, a commit SHA, a lockfile path that moved. Check the hit rate, not the configuration.

Stages running in sequence that have no dependency on each other. Lint, unit tests and image build frequently run one after another purely because that is the order somebody added them. Nothing links them. Running them in parallel is often a ten-line change.

Image builds that invalidate on the first layer. A COPY . . before the dependency install means every commit — including a README change — rebuilds every layer. Reordering the Dockerfile so dependencies are copied and installed before application code is usually the single largest win available, and it takes minutes.

Queue time. If a third of your total is spent waiting for a runner, no amount of stage optimisation helps. This one is invisible in most dashboards because it happens before the job starts. Look for it specifically.

Rank by minutes saved per hour of work

With stage numbers in hand, the change list orders itself. Estimate the hours each fix takes, divide the minutes saved, and do them in that order.

It is normal for the top two changes to account for most of the total gain. It is also normal for at least one of them to be a configuration line rather than an architectural change — which is exactly why measuring first matters. The expensive rewrite someone proposed in the first meeting is rarely in the top two.

Then measure again

Re-run the same thirty-run sample after the changes. Partly to confirm the gain is real, and partly because the bottleneck moves: fix the cache and queue time becomes the largest number. Pipelines do not get optimised once.