A few simple tricks for building chatbots | Christopher Krapu %F0%9F%8C%81"> Contents

Thanks to LLMs capable of reasoning and using 1M+ tokens in a single prompt, the simple chatbot has grown up. Now it is a proper agent with virtually all the digital capabilities of its human counterpart.<br>That said, hooking up the OpenAI API to a simple back-and-forth conversational system is still the entry point for AI engineering.<br>Even though 99.9% of this system’s complexity is abstracted away behind the inference API, there are a few concepts that I see people take a while to grasp as they learn how to build systems on top of AI.<br>This short note discusses a few of them. I haven’t found good sources or references for these ideas elsewhere, so I hope this note is useful.<br>Efficient citation via clipboarding<br>LLM-driven search à la Perplexity may seem like a 2024 thing, but it is probably still one of the top three most valuable use cases for AI. In this workstream, being able to reference sources and efficiently show them to the user is critical.<br>Many starter kits for chatbots (e.g., Chainlit and Open WebUI) can easily render Markdown. Markdown allows you to write hyperlinks with a string like Please click me!, which is easy to write and simple to remember.<br>LLMs are perfectly capable of writing hyperlink strings like these and require only a few short prompt instructions to do so. However, asking the LLM to write the hyperlink comes with a few limitations:<br>While the user is waiting for the text to finish streaming, they will see partial hyperlink strings like [Please click me!](https://www.yo... Once the string is complete, it will be rendered, causing the partial text to disappear and the hyperlink to appear in its place. This jump is irritating and creates a poor user experience.<br>As system complexity and scope grow, you will likely begin citing sources with very long URLs that take a long time (>100 ms) to write token by token.<br>This approach is token-inefficient because the same result can be produced more cheaply.<br>The solution is simple. As the reader, you have probably already implemented it or reinvented it on the spot.<br>Regardless, here it is: instruct the LLM to write short citation keys like @d3 or #a1. A postprocessor sitting on top of the text stream then captures these keys and deterministically replaces them with full hyperlinks. I personally call this the “clipboard” approach in all of my codebases.<br>Some caveats do apply. Namely, you need some mechanism to give the source a nice descriptor (like “Annual Shareholder Report, 2026”). The naive approach from earlier lets you prompt the LLM to create the hyperlinked text on the fly; this approach doesn’t let you do that out of the box.<br>This approach also separates citation styling from prompting logic. That separation is a good idea (IMO), but it can be a pain if you’d rather just write prompts instead of extending application logic. Visually, here’s what it looks like:

You can also generalize this approach to any string that the LLM might generate inefficiently. If your LLM is powering an agent with tools for accessing APIs or reading files in a structured format, users may ask questions whose answers require the model to regurgitate data nearly verbatim. If the source is a long CSV or JSON file, having the LLM copy its contents token by token is wasteful. Use the clipboard approach to copy and paste the output instead.<br>Your choice of placeholder token is mostly unimportant, so long as it is small and unlikely to show up in the generated text for unrelated reasons. Avoid using standard formats like [1] or [2], since hallucinating LLMs will typically generate false citations in those formats. For richer styling, you could probably use XML or a similar format.<br>Constructing follow-up questions or actions<br>Most domain-specific AI platforms (e.g., Microsoft Copilot, Perplexity, and Glean) now supply recommended or follow-up questions for users to click. This is partly a gimmick from product managers to juice usage numbers, but it can sometimes be genuinely useful.<br>The best follow-up questions (e.g., “Would you like me to also search for restaurants closer to you?”) know both 1) what the user originally asked and 2) how the system responded in the most recent turn. The standard approach I saw in a number of systems from 2025 and earlier was to run a specific prompt to make the LLM generate these questions.<br>Again, this is wasteful and inefficient. Nowadays, using structured generation plus streaming, you can have an LLM answer the user’s question and generate the follow-ups in a single prompt using a schema like the following:<br>(show a schema that uses the OpenAI API to answer and generate follow-ups)

The combined approach can be cost-neutral, as most of the instruction tokens from the follow-up prompt are moved into the main response prompt. In either case, the instructions should remain in the prompt cache 100% of the time, so the prefill cost of additional...