A developer pushes a change and goes to lunch. Two and a half hours later the pipeline is done — if it didn’t get cancelled along the way, if nobody pushed again behind them, if a flaky test didn’t trip the retry loop and turn the build into something that’ll finish tomorrow.
This was the team’s normal. Shipping anything meant waiting for the pipeline. Shipping urgently meant going around it — and finding out whether your hotfix worked the same way your users did. The runner bill climbed. The Slack channel filled with “is CI done yet” the way other teams’ channels fill with memes.
They knew it was bad. They thought the answer was more parallelism, bigger runners, more retries. We were brought in to find a different answer.
What we found
The pipeline wasn’t slow. It was wasteful — repeating work that had already been done, retrying work that was never going to pass, running work nothing was waiting on.
The base image was rebuilt every run. OS packages, internal repos, users, permissions — all of it static day to day, all of it reinstalled on every build. We pulled the image build out into a daily cron job. The pipeline now pulls a pre-baked image and gets to work.
Docker layer caching wasn’t on. Every layer was rebuilt from scratch even when nothing in it had changed. Turning caching on cut what was left of the build roughly in half — the layers that change get rebuilt, and the rest are pulled from cache the way they should have been all along.
The unit-test concurrency had grown into a monster. This one is worth the long version, because it’s instructive.
The team had tried to parallelize their tests. The first attempts failed under concurrency — the usual story, shared fixtures, shared ports, shared state. Reasonable instinct: maybe we just have too many workers. So they built a fallback. Run with six workers; if any test failed, retry with three; if that still failed, retry with one. The intent was recovery. The effect was a tax on every failed run.
Their suite is a little over 9,000 tests. If the 9,000th test fails, the runner spends the next two hours retrying the entire suite with three workers, then again with one. Failures — which are the normal, useful signal of CI; you want them, you want them fast — became the most expensive thing the pipeline could do. Breaking a test cost the team an afternoon.
The fix was structural, not parametric. Real test isolation removed the concurrency failures the fallback was trying to paper over. We deleted the fallback. We reorganized the test files so they could be sharded properly and split the suite into eight shards that run in parallel.
There were dead steps left over from old infrastructure. Setup for package repositories the team had stopped using a year ago. Cheap to find, cheap to delete.
None of these moves are exotic. Most of them are in the GitHub Actions docs. The work wasn’t inventing the fixes — it was finding them in this pipeline, and doing the structural work that let the easy ones land safely.
What changed
A developer can now push a change and trust that the pipeline will be done in around half an hour. Hotfixes ship the same afternoon they’re written. Cancellations went to zero in the week we measured after — not “down significantly,” zero. The seven-hour outliers stopped happening. The runner bill dropped. The “is CI done yet” pings stopped.
The team didn’t need more infrastructure. They needed the infrastructure they already had to stop fighting them. This is what Platform & automation usually looks like.