A reverse mullet type of internet

While updating my personal site to reflect my post-Ph.D. career direction, I ran into a basic but annoying question: who was the site actually for? My work is scattered across projects, papers, platforms, and terse academic writing, and I am not especially good at self-promotion. After feeding my career and projects to Claude, neither I nor the llmunculus could figure out who should be reading my site. So after a while of back and forth, it helpfully suggested the AI technique of ‘putting a bunch of stuff into /llms.txt.’ . What follows in this text is a brief story of my wrasslin' with LLMs, a pseudoexperiment in LLM behavior, and a shot-in-the-dark into the future of the internet.<br>Don't bet money on anything I write here.<br>Having been handed this easy copout from a problem that deserved careful attention, I realized two things. First, people don't like to read.[1] Second, as more people interact with the web through LLMs like Claude, ChatGPT, or Grok, my site no longer needs to be generic enough to serve every audience, or specific enough to serve a particular one. Instead, I could just put all the information somewhere LLMs know to look, while keeping the human-facing page a clean overview of the basics plus a few selected projects I think are cool. A more inquisitive audience would have their specific interests catered to by the LLMs, which filter the information and serve the user only with the parts that he actually cares about.<br>I was curious about whether or not this was even worth it, if llms even bothered to pull the llms.txt file. So as a quick vibe check, I spun up 6 models that ranked highly on the openrouter benchmarks[2], and prompted each model 10 times with the prompt "Explore spock.is and tell me everything you can find". The prompt was chosen specifically to 'encourage thoroughness', without explicitly priming the models to fetch the llms.txt file (I don't love the amount of meta-cognition I find myself doing when working with LLMs).<br>Modelllms.txt fetchedclaude-opus-4.80/10claude-opus-4.70/10gpt-5.51/10deepseek-v4-pro0/10qwen3.7-max0/10grok-4.200/10Results without nudgeInitially, only openAI's gpt-5.5 knew to proactively check for a llms.txt and only did so once, but none of the other models actually ever checked for it. Calling on years of playing around with prompt injection, I figured that I could probably 'encourage' the models to read the information with a helpful nudge. So I placed a discreet link to the llms.txt next to the links to my socials.

llms.txt<br>After adding the nudge to the llms.txt file, I re-ran the 'experiment' with the same prompt as before, and the results were very different. Every single model hit the llms.txt file in all 10 runs.<br>ModelWith nudgeclaude-opus-4.810/10claude-opus-4.710/10gpt-5.510/10deepseek-v4-pro10/10qwen3.7-max10/10grok-4.2010/10Results with nudgeThe lesson: if you want LLMs to fetch llms.txt, you have to advertise it. Obvious, maybe, but it does indicate that if you want to have a good llms.txt, the content isn't enough, you also have to advertise it.<br>But, anyway, this, I felt, was fantastic. I now could comfortably cram all my personal information into llms.txt, and fill my designed-for-humans site with surface-level information, and all the pretty effects that I have come to think of as digital 'key-janglers'. However, as I was cramming in all this information, I started worrying, like many over-engineering Ph.D.'s would, that the sheer weight of my accomplishments would produce an llms.txt that, when pulled, would overflow the context window of the models requesting it. Mine was about 1900 tokens, so... probably not in my case. But, many agentic systems employ a sort-of summariser model in between them and the pulled documents, so a large and impressive llms.txt might end up summarized or simply missing vital information that the summariser didn't know to pass along to the main model.<br>This problem felt familiar, and easily solvable. RAG[3] systems are a well known solution, allowing LLMs to query for specific information and receive only the relevant chunks. So I added two endpoints to my website (requiring only a modest total rewrite): /llms/?query= and /llms/json?query=, both only ever advertised to LLMs, each accepting a simple string query.<br>I was skeptical if the LLMs would even call the endpoint even when nudged to, since models have guardrails in place to prevent behavior-altering commands such as prompt injections or jailbreakings (remember DAN?). But, I updated the nudge to explain these new endpoints (and include the size in tokens of my llms.txt), and ran the experiment again, this time running each model 30 times.<br>llms.txtTurns out, not only do the models query random end points that promise information, but they do so eagerly. DeepSeek in particular making on average 7 queries each run. The models seemed to prefer plaintext over structured json, and every single model's first instinct was to pull...