tezcatl: a 2MB alternative to Puppeteer for scraping on macOS • George MandisI've been working on a small project that uses whereami and nearme to do some local scraping and experimentation with LLMs to build enriched datasets. The details of that project are for a future post, but the short version is I needed to fetch web pages, extract some data and move on.<br>A quick cut to the chase:
If you're on a Mac and find yourself needing to do light scraping work on websites as they are rendered—not as they are sent over the wire—please consider trying tezcatl:<br>https://github.com/georgemandis/tezcatl<br>Some selling points:<br>A little like Puppeteer but only ~2MB instead of nearly 300MB.<br>Built around WebKit, which is already on your Mac, and only cares about returning an accurate snapshot of the DOM.<br>About as lightweight as a tool like this can be; has no dependencies or build steps you'll ever have to think about.<br>Can become a tool in your scraping + parsing toolkit right alongside jq, curl and other CLI affordances you reach for daily.<br>For those that want more context and story, read on!
The problem<br>curl handles about 70% of what I need. Most pages serve a usable version of their content in the initial HTML response and you can pipe curl into whatever you want—namely Simon Willison's llm tool in this case, which I adore.<br>But some sites render everything client-side with JavaScript and come back as soulless husks. A and a pile of tags—an illegible affront to a medium inherently centered around reading and the written word. Oh, the horror. The horror.<br>The standard answer to this reality is Puppeteer or Playwright. Spin up a headless Chromium, wait for the page to render, grab the DOM. It works, but it's a lot of overhead for what I actually needed, which was just to load a URL, wait until the page loaded and grab the HTML.<br>I didn't need cross-platform, cross-browser QA automations, screenshots or HAR dumps. I just needed the rendered DOM so I could strip the tags and ask an LLM to do some things with the actual words.<br>Every Mac ships with WebKit. It's the same engine Safari uses. Apple exposes it through WKWebView, which is how native apps embed web content. You can use it from a CLI tool.<br>So I made tezcatl. It's not for automating browsers or running tests. It's just for scraping web pages that don't render without JavaScript.<br>$ tezcatl https://example.com<br>Example Domain...
$ tezcatl https://spa-site.com --wait=2000<br># waits 2s after load for JS to render<br>It creates an offscreen WKWebView, loads the URL, waits for the navigation delegate to fire, optionally pauses for additional JS settling time, then evaluates JavaScript against the page (if specified) and writes the result to stdout. By default it returns the full rendered DOM. With --eval you can run arbitrary JS instead:<br>$ tezcatl https://example.com --eval="document.title"<br>Example Domain
$ tezcatl https://example.com --eval="document.querySelectorAll('a').length"<br>A real example<br>Apple's own developer documentation (at the time of this writing) renders in the client and seems to be a Vue app. curl gives you this:<br>This page requires JavaScript.<br>With tezcatl you can render the page and pull structured data out of it:<br>$ tezcatl https://developer.apple.com/documentation/ --wait=3000 \<br>--eval="JSON.stringify([...document.querySelectorAll('a.card')].slice(0,5).map(c => ({<br>title: c.querySelector('.title')?.textContent?.trim(),<br>description: c.querySelector('.card-content .content')?.textContent?.trim(),<br>url: c.href<br>})), null, 2)"<br>You'll get something like this:<br>"title": "Explore the new design principles",<br>"description": "Learn how to design and develop beautiful interfaces that leverage Liquid Glass.",<br>"url": "https://developer.apple.com/documentation/TechnologyOverviews/liquid-glass"<br>},<br>"title": "Adopting Liquid Glass",<br>"description": "Find out how to bring the new material to your app.",<br>"url": "https://developer.apple.com/documentation/TechnologyOverviews/adopting-liquid-glass"<br>It pipes with other tools, which was another reason I rolled my own solution. I wanted something that fit into the CLI workflows I was already building with whereami, nearme, lingua and loupe.<br># Get the rendered DOM and extract text with lingua<br>tezcatl https://example.com | lingua detect
# Scrape a title for use in a script<br>TITLE=$(tezcatl https://example.com --eval="document.title")
# Find the nearest pizza place, grab its website, render it<br>whereami --json | nearme "pizza" --json | jq -r '.[0].url' | xargs tezcatl<br>How it's built<br>Like the rest of my Zig tools, tezcatl talks to the Objective-C runtime directly. No Swift or Objective-C source files. Zig calls objc_msgSend and friends to create a WKWebView, register a navigation delegate class at runtime with objc_allocateClassPair, and wire up the completion handler using the ObjC block ABI.<br>WebKit's evaluateJavaScript:completionHandler: expects the completion handler to be an Objective-C block—not a function pointer. Blocks have a...