The scraper bugs that don't throw: six silent failures from building production scrapers · GitHub

/" data-turbo-transient="true" />

Skip to content

-->

Search Gists

Search Gists

Sign in

Sign up

You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.

Dismiss alert

{{ message }}

Instantly share code, notes, and snippets.

ConorsCode/the-scraper-bugs-that-dont-throw.md

Created<br>July 30, 2026 08:08

Show Gist options

Download ZIP

Star

(0)

You must be signed in to star a gist

Fork

(0)

You must be signed in to fork a gist

Embed

Select an option

Embed<br>Embed this gist in your website.

Share<br>Copy sharable link for this gist.

Clone via HTTPS<br>Clone using the web URL.

No results found

Learn more about clone URLs

Clone this repository at &lt;script src=&quot;https://gist.github.com/ConorsCode/ea7dbfa4e389554c16c308d0a19b67b2.js&quot;&gt;&lt;/script&gt;

" readonly="readonly" data-autoselect="true" data-target="primer-text-field.inputElement " aria-describedby="validation-1c338a1e-f273-4f30-ba3b-88bce827c5d2" class="form-control FormControl-monospace FormControl-input FormControl-small rounded-left-0 rounded-right-0 border-right-0" type="text" name="gist-share-url-sized-down" />

Save ConorsCode/ea7dbfa4e389554c16c308d0a19b67b2 to your computer and use it in GitHub Desktop.

Embed

Select an option

Embed<br>Embed this gist in your website.

Share<br>Copy sharable link for this gist.

Clone via HTTPS<br>Clone using the web URL.

No results found

Learn more about clone URLs

Clone this repository at &lt;script src=&quot;https://gist.github.com/ConorsCode/ea7dbfa4e389554c16c308d0a19b67b2.js&quot;&gt;&lt;/script&gt;

" readonly="readonly" data-autoselect="true" data-target="primer-text-field.inputElement " aria-describedby="validation-7f10f24a-e373-4689-8684-e18caf7ae818" class="form-control FormControl-monospace FormControl-input FormControl-small rounded-left-0 rounded-right-0 border-right-0" type="text" name="gist-share-url-original" />

Save ConorsCode/ea7dbfa4e389554c16c308d0a19b67b2 to your computer and use it in GitHub Desktop.

Download ZIP

The scraper bugs that don't throw: six silent failures from building production scrapers

Raw

the-scraper-bugs-that-dont-throw.md

The scraper bugs that don't throw

Every scraper I've written has failed loudly at some point, and those failures were never the problem. A 403 is easy. A timeout is easy. A selector that returns null blows up on the next line and you go fix it.

The ones that cost me real time all had the same shape: the run finished, the logs were clean, the exit code was zero, and the data was wrong or absent. Nothing to grep for. You only notice days later when you look at a row and think, that can't be right.

Here are the ones I hit most recently, building a couple of scrapers for lu.ma and for job boards. None of them threw an exception. Every one of them produced a "successful" run.

1. A redirect that silently empties your crawl

lu.ma 301-redirects to luma.com. Harmless-looking. But if you're using Crawlee and enqueueing discovered links the obvious way:

await enqueueLinks({ selector: 'a[href^="/"]' });

...you get nothing. Not an error. An empty dataset from a crawl that reports success and finishes suspiciously fast.

The reason is that enqueueLinks defaults to a same-hostname strategy. You started on lu.ma, you got redirected to luma.com, and now every link you discover is on a different hostname than the one Crawlee is comparing against. All of them get dropped. Quietly, because dropping off-domain links is normal behaviour, not an error condition.

The only way I saw it was turning on debug logging and finding this repeated once per link:

does not match the enqueue strategy (same-hostname)

The fix is to stop asking enqueueLinks to make that judgement and add the URLs yourself:

await crawler.requestQueue.addRequests(<br>urls.map((url) => ({ url })),<br>);

What makes this class of bug nasty is that "zero results" is indistinguishable from "the site had nothing to give you". If you don't know how many rows to expect, you can't tell the difference. Which leads to the general lesson at the end.

2. A price field with three different types

lu.ma server-renders its whole page state into __NEXT_DATA__, which is lovely — no browser needed, just parse the JSON. Inside it, each event has a ticket_info.price.

Sometimes that's a number. Sometimes it's a string. And sometimes it's this:

{ "cents": 2500, "currency": "usd", "is_flexible": false }

If your code does the natural thing:

priceDisplay = String(ticketInfo.price);

then a portion of your dataset now says [object Object]. No exception, valid string, wrong answer. I found it by eyeballing output from a live run, which is not a strategy.

3. Two fields that contradicted each other

Same site. There's...