The poisoned pill that killed Greppr indexing | lead > prompt #

In my last podcast episode, I spoke about the various technical lessons I have learned from running a web search engine, in particular the we crawling and indexing components that I believe to be the most difficult parts to maintain. As soon as I published that episode, I hit another major issue that I will document here now.

The poisoned pill

The backend of Greppr is an Apache Solr cloud cluster, with a swarm of custom indexers that I wrote myself. When I woke up this morning, my inbox was full of Monit alerts that the Solr index had stopped growing, a sure sign that the index swarm was broken. Not a great start to the day!

My crawler error log was full of the follow errors, thousands of times over:

"2026-07-29 08:13:43",ERROR,CrawlTask,"[worker 1432746] Solr HTTP error: OK (500)

""responseHeader"":{

""rf"":1,

""status"":500,

""QTime"":0

},

""error"":{

""errorClass"":""org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException"",

""msg"":""bytes can be at most 32766 in length; got 44043"",

""trace"":{

...

In plain English, Solr has been handed a URL of length 44043 to index, but that exceeds its allowable max length of 32766 (32KB) characters. Why such a long URL exists on the Internet I have no idea, but as I mentioned in my previous podcast the Internet is wild and standards are usually ignored, so you have to build your crawler and indexer stack defensively, which is exactly that I didn't do in this instance!

Rather than gracefully handling this error, my worker thread instead freaked out and left the bad URL in the queue for the next worker to collect, resulted in my 40 worker threads all being locked for several hours on a single bad URL, yikes. Searches could still be performed on the frontend, but no new URLs were added to the backend index during this time.

The death loop happened because the worker crashed before it could mark the URL as completed, so the URL got recycled back into the queue, only to trigger the same trap again and again...

The fix

The fix is simple but requires validating not just the URLs in your queue, but the data you extract from the wild web. In my case, the 44KB string was hiding in a badly formatted canonical tag inside the HTML! Here is some pseudo-code showing how we enforce the limit on extracted elements:

// Protect Solr from Lucene's 32KB term limit<br>canonical_link = extract_canonical(html)

if (len(canonical_link) > 8192) {<br>// Log the error and discard the garbage canonical tag<br>log("Canonical link exceeds 8KB limit. Ignoring.");<br>canonical_link = ""; // Fallback to the safe, original URL

As you can see, I decided to go with 8KB as a reasonable URL limit rather than 32KB. Anything longer gets discarded.

The amazing this about this Greppr project is that I am constantly learning new things, not just about system architectures but also about how broken the Internet actually is! A huge amount of the content I crawl is just discarded because the quality of the data is so low.

"the Internet is dark and full of terrors."

Melisandre (probably)

© 2001-2026 John Collins. Follow for updates: