2026-06-30: CI/CD pipelines not being triggered on merge requests. (#22412) · Issues · GitLab.com / GitLab Infrastructure Team / Production · GitLab
2026-06-30: CI/CD pipelines not being triggered on merge requests.
# CI/CD pipelines not being triggered on merge requests. (Severity 1)
**Problem**: CI/CD pipelines on merge requests were not being triggered or were experiencing major delays for all customers on GitLab.com due to Sidekiq CPU saturation and a backlog on the urgent-ci-pipeline shard.
**Impact**: All customers on GitLab.com experienced significant delays or failures triggering CI/CD pipelines on merge requests. Pipeline status updates were outdated, jobs were deferred, and many users could not run CI/CD workflows. At peak, the backlog in the urgent-ci-pipeline queue reached over 800,000 jobs. Pipelines are now being created successfully, but some jobs are still processing with a delay as the system works through the remaining queued work. Monitoring shows throughput and processing rates have returned to normal, though some minor delays persist.
**Causes**: A single user created a large number of merge requests, some with over 52,000 commits, which saturated the `urgent-ci-pipeline` Sidekiq shard and created a backlog. Even after blocking this user and deleting their queued jobs, the backlog persisted due to high overall demand and concurrency throttling for `MergeRequests::CreatePipelineWorker`. No individual user is now dominating the queue.
**Response strategy**: We blocked the user responsible for the initial backlog at 09:45 UTC and deleted their queued jobs. We expanded job deletions to other queues and then increased the Sidekiq `urgent-ci-pipeline` worker capacity from 1200 to 1500 replicas at 10:56 UTC. This scaling cleared the backlog, restored normal pipeline creation, and reduced job deferment and worker saturation. Some jobs are still processing with a delay as the system catches up, but overall rates are back to normal levels as of 11:22 UTC (Etc/UTC).
_This ticket was created to track_ [_INC-11596_](https://app.incident.io/gitlab/incidents/11596)_, by_ [_incident.io_](https://app.incident.io) 🔥
issue