Rebuilding Cognition's Agentic MapReduce | Matt Rickard

Matt Rickard<br>About Posts Subscribe

Rebuilding Cognition's Agentic MapReduce<br>Jul 3, 2026<br>How do you run large-scale agent tasks across a codebase? Today there's three approaches.<br>One thread . Compaction removes detail. Bad explorations poison future context. Simple and cache-efficient but slow.<br>Manager thread, subagents . Increases available context and can be faster. Subagents share the same view of the codebase, so can't try things in parallel. Worktrees give each subagent its own sandbox, but you run into single-machine resource contention (ports, disk, RAM).<br>Subagents across machines . Largely unsolved. How do you shard agents? How do they communicate? How do you aggregate output?<br>Cognition is trying to solve it for security with their Security Swarm. It finds vulnerabilities in large codebases using a technique they call Agentic MapReduce . In their eval, they found more vulnerabilities and ~30% lower cost than other agentic solutions.<br>An agent builds a security profile and threat model for the codebase and uses this to shard the workers. A swarm of agents works in parallel on the shards and reports security findings. A reducer session consumes the workers' findings and composes attack chains across shards to elevate CVEs. A verifier fans out again over the findings to confirm attack vectors.<br>Cognition's version is security-specialized end-to-end. I wanted the same loop for more generalized tasks: research, coding, and review. That meant rebuilding each stage without task assumptions. So I reverse-engineered their eval dataset and tested my approach. I ended up with something that works locally, but can also scale to much larger cloud deployments.<br>The full skill is here so you can test a small version yourself, but here's the simple algorithm.<br>Map. Subagents are initially sharded across the codebase by folder. Each worker gets its own git branch.<br>Reduce. One iteration is complete when all agents have committed to their branches (or timed out). A reducer agent reads each branch, dedupes and merges compatible findings. It can also run verification checks. It records the result as a new candidate branch.<br>Loop. The next mapping agent decides how to allocate workers. They can continue from the candidate branch, continue from a worker branch from the last generation, or start fresh from a new folder shard.<br>Git commits and branches are the primary agent-to-agent communication method. Git merge powers the reduce step. Agents natively know how to use the system: they know how to inspect past work via branch history, merge and consolidate changes, and commit their work thanks to decades of git training data.<br>Using git as the primitive has other useful properties:<br>Fault tolerant. Agents resume from stopped or completed jobs. Just checkout the latest candidate branch.<br>Auditable. Every change is auditable via git commits. Adding chat context to the commits makes the transcript auditable as well. Trace changes back to the exact worker and iteration they were born via branches.<br>Generalizable. Run any task through this pipeline. It makes no assumptions about your context. Research tasks can write findings as files. Code modification tasks can edit the files normally. There's no special infrastructure that's needed to run the simple version. Run any number of workers, any number of iterations.<br>I ran 13/34 repositories in the benchmark before I hit my rate limit. Three iterations for each. The average cost per run was $26.70. The only instruction was "Scan this repository for security vulnerabilities" plus a bit of bookkeeping.<br>6 of 13 found the target CVE exactly<br>2 found other confirmed CVEs but not the target<br>5 missed the target but surfaced plausible findings with no confirmed public CVE.<br>Patterns I observed:<br>The reducer does real synthesis work. In dex , the first reducer step combined work from the three worker shards to find the target vulnerability, GHSA-7qjx . The second mapper spawned a worker to verify it. In koel , iteration one found adjacent vulnerabilities but not the target. The second iteration found it and the third verified it, matching GHSA-7j2f . filebrowser did not find its target, but surfaced a symlinked-scope escape ( CVE-2026-54094 ) in iteration 2 and a copy/rename path traversal ( CVE-2026-32758 ) in iteration 3. Swarms were not always helpful. smallbitvec and archive were found on the first pass by every worker.<br>Three workers over 3 iterations created 9 worker branches and 3 candidate branches, ~12 refs per repo, and 150+ across the 13-repo benchmark. Run this 5x5 on PRs and you'd create 25 extra branches per PR.<br>Git was a helpful substrate: the agents knew it, and it gave communication, fault tolerance, and audit for free. Where should these branches live? The workers need scoped credentials and a durable remote with cheap repos. GitHub isn't the right answer: it treats branch refs as artifacts for humans, not agents. Corigin is the...