Sleeping at Scale | Buildkite<br>/vibe_checkOpinion9%Story16%Lessons22%Craft25%Code28%

Article composition: Code 28%, Craft 25%, Lessons 22%, Story 16%, Opinion 9%

Sleeping at Scale

DateAug 19, 2026<br>AuthorJosh Deprez<br>AnimatorsJulián Pinzón EslavaTom Watts

Read time15 min read<br>AgentsCopy for LLMsView as markdown

ShareTwitter/XLinkedIn

{label}

-->

One Buildkite Agent doing something once per second (like collecting new log output, splitting it into chunks and preparing them for upload) is not especially interesting. But there are hundreds of thousands of Buildkite Agents connected and running around the world talking to our backend. If enough of them happen to do the same thing at roughly the same time, a harmless little loop becomes a large spike in work for the servers on the other end.

We need the agents to keep a regular pace without marching in step and this has turned out to involve more thought and more kinds of loops than we expected.

To understand what we’ve done to achieve this, we need to take a step back and go back to basics: one program that needs to do a thing over and over again, forever.

Doing a thing repeatedly forever

The simplest way to write that (in Go, but this is hopefully readable to everyone) is:

go

for {<br>doThing()

Immediately repeating the action without any sleep in between will probably cause some resource to be consumed as quickly as possible, such as CPU or network bandwidth.

So let’s add a sleep to slow things down:

go

interval := 1 * time.Second<br>for {<br>doThing()<br>time.Sleep(interval)

Do the thing, sleep 1 second, repeat.

Question: In the code above, is doThing called exactly once every second?

Interactive animation description: A timeline shows doThing running for 300 milliseconds, followed by a full one-second sleep. Successive executions begin 1.3 seconds apart rather than once per second because each interval includes both the work and the sleep.time.Sleep() 1.00s · doThing() 300ms<br>Play▶<br>Time

0s

1s

2s

3s

4s

5s

ExecutingSleeping<br>300ms0.00sone iteration takes 1.30s

Answer: No. Aside from the system clock changing, drifting, or otherwise just being inaccurate, the loop does not account for the length of time it takes doThing to do its thing.

Waiting 1 second in between actions leads to the actions happening less frequently than once per second.

This approach was used in the Buildkite Agent (repo) for processing job logs not that long ago!

What is the Buildkite Agent?+The Buildkite Agent is a small, cross-platform build runner. It polls Buildkite for work, runs jobs on Buildkite-hosted or self-hosted infrastructure, streams each job’s logs and status back to Buildkite, and uploads its artifacts. It is the worker program responsible for running CI jobs.<br>In the example discussed here, doThing() roughly corresponds to collecting the latest job output, splitting it into chunks, and preparing those chunks to be uploaded to Buildkite.

Let’s be a bit more sophisticated and use a time.Ticker, Go’s recurring timer. time.Tick gives us a stream of ticks, one per interval (e.g. 1 second). If our code falls behind (i.e. doThing takes longer than the tick interval), Go may skip ticks rather than queueing every tick we missed.

go

interval := 1 * time.Second<br>for range time.Tick(interval) {<br>doThing()

Interactive animation description: A timeline shows the loop waiting for the first ticker event, then running doThing at 1, 2, 3, and 4 seconds. Each execution takes 300 milliseconds and leaves 700 milliseconds before the next scheduled tick, so execution time does not accumulate between starts.time.Tick() 1.00s · doThing() 300ms<br>Play▶<br>Time

0s

1s

2s

3s

4s

ExecutingWaiting<br>for tickWaiting for tick<br>one iteration takes 1.00s

The ticker’s one-second schedule keeps advancing while doThing() runs. Here, 300ms of work leaves 700ms before the next tick, so each execution still starts 1 second after the last. If the work takes longer than the interval, the ticker cannot keep the loop on time.<br>The compact for range syntax above waits for each tick and then runs the body of the loop. We can write those same steps explicitly: first store the stream of ticks in tick, then use to wait for the next one:

go

interval := 1 * time.Second<br>tick := time.Tick(interval)<br>for {<br>tick<br>doThing()

This is the same as above. But both implementations have the disadvantage that on the first iteration, we wait for the interval up front before calling doThing. In the time.Sleep approach, we did the thing and then waited.

This is easy to remedy: we can reorder the operations inside the loop.

go

interval := 1 * time.Second<br>tick := time.Tick(interval)<br>for {<br>doThing()<br>tick

Now it will do the thing, wait until the next second, do the thing, wait until the next second… and the ticker can keep things as close to on-time as possible, regardless of how long doThing takes to run. That solves it, right? (Right?)

Interactive animation description: A timeline shows doThing running immediately at 0...