Supervised fire-and-forget in Go | Redowan's ReflectionsSkip to content
These days, unmanaged go func() calls don’t appear as often as they used to in the early<br>days of Go.<br>At this point, everyone knows Dave Cheney’s maxim to “never start a goroutine without<br>knowing how it will stop”. Even so, I still occasionally run into unsynchronized<br>go func() calls doing fire-and-forget work.<br>I’m not talking about jobs handed to a dedicated task queue such as Asynq. In that case,<br>the task system owns the lifecycle of the tasks it starts. I mean smaller, best-effort jobs<br>where it’s super tempting to start a task with go func() and just forget about it. Sending<br>a notification or writing an expensive diagnostic log often falls into this category.<br>It typically looks like this:<br>func (h *handler) createOrder(w http.ResponseWriter, r *http.Request) {<br>user := r.URL.Query().Get("user")<br>ctx := r.Context()
go func() {<br>h.tasks.SendNotification(ctx, user) // (1)<br>}()<br>go func() {<br>h.tasks.WriteDiagnosticLog(ctx, user) // (2)<br>}()
w.WriteHeader(http.StatusAccepted)
In the handler above:<br>(1) sends a notification in a background goroutine<br>(2) writes a diagnostic log in another background goroutine<br>The response doesn’t wait for either one. On a long-running server this looks reasonable, as<br>main will most likely outlive both calls. But this suffers from a few issues:<br>every request can start two more goroutines, with no limit on active work<br>both tasks get r.Context(), which net/http cancels when the handler returns. If the<br>jobs are cancellation aware (ideally they should be), they can bail before they’re done<br>a panic in either goroutine crashes the whole process, because nothing recovers it<br>main has no way to wait for either task during shutdown. A restart kills whatever is<br>still running<br>Every task goes through one worker pool #<br>One fix I’ve been using is a small worker pool backed by a buffered channel:<br>each background task is sent to the channel as a func() closure<br>workers started at the composition root, usually main, drain the channel and run the<br>closures<br>the number of workers is fixed, so concurrency stays bounded<br>nothing else in the application starts a background goroutine. All background work goes<br>through this pool<br>at shutdown, main closes the channel<br>In its simplest form, it looks like this:<br>// Make a buffered channel in main.<br>tasks := make(chan func(), 64)
// Still in main, start a fixed set of workers to drain<br>// the channel and run the tasks.<br>for range 4 {<br>go func() {<br>for task := range tasks {<br>task()<br>}()
// From a handler, or anywhere else, send a task as a closure.<br>tasks func() {<br>doWork()
// At shutdown, close the channel.<br>close(tasks)
Each receive loop runs one task at a time, so four workers mean at most four tasks run at<br>once. The buffer holds 64 pending closures. Once it fills, the send blocks until a worker<br>receives a task. Closing the channel ends the receive loops. The workers drain whatever is<br>left in the buffer and exit (not shown here).<br>A real pool needs more than a channel #<br>While the simple form above tackles the bounded concurrency issue, it doesn’t handle a few<br>things that we need in a real worker pool:<br>error handling: no caller waits for a result, so failures are handled inside the task. A<br>panic must not crash the server<br>context propagation: a task must not inherit the request’s cancellation, but it may still<br>need the request’s values<br>timeouts: once a task is detached from the request context, nothing cancels it. It needs<br>its own deadline<br>graceful shutdown: after a shutdown signal, every accepted task should finish before<br>main exits<br>The following API aims to cover all of these requirements.<br>The entire API is three functions #<br>The complete implementation is a small bg package that wraps the channel in a struct<br>called Background:<br>// internal/bg/bg.go<br>type Background struct {<br>tasks chan func()<br>workers sync.WaitGroup<br>onPanic func(any)
mu sync.RWMutex<br>stopped bool
func New(capacity, workers int, onPanic func(any)) *Background<br>func (b *Background) Submit(task func()) bool<br>func (b *Background) Stop()
New builds the pool, Submit queues a task, and Stop shuts the pool down. The<br>constructor makes the channel and starts the workers (argument validation elided):<br>// internal/bg/bg.go<br>func New(capacity, workers int, onPanic func(any)) *Background {<br>background := &Background{<br>tasks: make(chan func(), capacity),<br>onPanic: onPanic,<br>for range workers {<br>background.workers.Go(background.work)<br>return background
workers.Go is WaitGroup.Go, added in Go 1.25. It starts each worker goroutine and tracks<br>it in the group in one call. Stop waits on that group during shutdown.<br>Each worker drains the channel:<br>// internal/bg/bg.go<br>func (b *Background) work() {<br>for task := range b.tasks {<br>b.run(task)
This is the receive loop from the sketch. It keeps pulling tasks until the channel is closed<br>and drained. The capacity and workers arguments bound the concurrency: at...