mycel: a search engine you can grow
skip to content
surface
· the soil line ·
01 · seed
It starts with one address.
Nothing is crawled that you didn't plant. mycel seed blog.example.org activates a host and enqueues its root. Links discovered off-host become candidates, recorded in the webgraph but never fetched, until you promote them.
$ mycel seed blog.example.org
02 · crawl
It knocks, and then it waits.
robots.txt is obeyed per RFC 9309. One request per host, ever, enforced by the claim query rather than by good intentions. A 429 doubles that host's delay, sticky, never lowered again. And it refuses to crawl at all until contact_url puts your name in the user agent.
(a live crawl log animates here)
03 · store
Pages are archived, not scraped.
Every fetch is appended to WARC, the same format Common Crawl publishes, and fsynced; only then does the same transaction advance the durable watermark. Kill the process mid-write: nothing corrupts, nothing orphans.
WARC is the source of truth. Everything else regrows.
04 · index
The index is a regrowable organ.
Text is extracted; near-duplicates are composted, exact by sha256, near by simhash. What survives enters a tantivy BM25 index. Corrupt it, upgrade it, delete it: mycel reindex grows it back from WARC.
10 Best Hikes Near Portland (2026 Edition, Updated)
10 Best Hikes Near Portland, 2026 Edition [Updated]
The 10 Best Hikes Near Portland, Updated for 2026
10 Best Hikes Near Portland (Updated 2026 Edition)
Best Hikes Near Portland: Top 10, 2026 Update
10 Best Portland Hikes: Updated 2026 Edition
10 Best Hikes Near Portland (2026 Edition, Updated)
7 fetched → 1 indexed · skip_reason: dup-near · simhash Δ ≤ 3
05 · rank
Well-connected soil ranks higher.
mycel computes harmonic centrality over its own host webgraph, offline, and folds it into every score: bm25 × (1 + 0.3 × centrality). Text relevance dwarfs link signals; centrality is a boost, never the ranking.
06 · search
Then it answers.
Phrases, site: filters, highlighted snippets. From your corpus, on your machine, in milliseconds. The daemon serves a UI and a JSON API on localhost.
Mycorrhizal networks and how forests share
https://fungi.example.org/nets
…fungal mycelium networks connect trees underground and move nutrients between them…
$ mycel search '"mycelium networks" site:fungi.example.org'
07 · federate
Everything you just watched is one node.
Peers dial each other by public key over QUIC. An allowlist is the only gate; trust is social, not cryptographic. Queries fan out and results interleave round-robin, so a peer's score can never buy rank. Corpora travel as sealed, blake3-verified WARC shards: no DHT, no tokens, no transitive flooding.
When both sides list each other, the network exists. Otherwise it simply doesn't.
instrument 01 the claim
One request per host is not a promise in the fetch code; it is a property of the SQL that hands out work. Watch four hosts get crawled: each lane holds exactly one token, and the politeness gate must drain before the next claim. Throttle a host and its delay doubles, sticky, never lowered.
the actual claim query (src/db.rs)<br>SELECT h.id, h.host, f.id, f.url, f.kind, f.attempts, f.depth,<br>h.robots_body, h.robots_fetched_at, h.crawl_delay_ms, d.sha256<br>FROM hosts h<br>JOIN frontier f ON f.id = (<br>SELECT f2.id FROM frontier f2<br>WHERE f2.host_id = h.id AND f2.state = 0 AND f2.next_attempt_at h.state = 1 AND h.in_flight = 0 AND h.next_fetch_at<br>ORDER BY h.next_fetch_at<br>LIMIT ?2
instrument 02 the watermark
A WARC member is appended and fsynced, and the same transaction that catalogs it advances shards.bytes, the durable watermark. So pull the plug. Go on. On reboot the shard is truncated back to the watermark: torn tails are unobservable, orphans are impossible.
instrument 03 the merge
Federated results interleave round-robin (local first, then each peer), deduplicated by URL. They are never re-sorted by score: BM25 scores from different indexes aren't comparable, so a malicious 99.9 buys nothing. Kill a peer mid-query; it contributes zero hits and zero delay.
1 binary
27 dependencies
~8,500 lines of rust
0 servers required
mycel
nutrition facts · per node
polite crawler RFC 9309
WARC archive source of truth
tantivy index disposable
harmonic ranking offline
HTTP API + UI localhost
federation off by default
contains 0 g :
JavaScript rendering
vector search
DHT
tokenomics
trustless peering
custom storage formats
telemetry
Grown from evidence.
The architecture is the product of adversarially verified research: 282 agents , three rounds, every load-bearing claim attacked by independent verifiers. 59 claims survived; 8 were refuted , and the refuted ones are logged so they can never silently re-enter the design.
The graveyard upstairs isn't decoration. It's section one of the research: the dominant failure mode of this project category is maintainer collapse and funding exhaustion, not algorithms. Every...