You 're right to push back | Andreas Nedelkos

You start out a new session and set the premise, ask some research questions, flesh out the architecture, and some hours later start building the first componenent. Maybe you have written everything in md files in high and low level detail. Or have TODO lists and agent.md files with clear, concise instructions. Invariably, a few hours hours later, the agent will make a decision you won’t like. Maybe it will take some weird initiative to create a class you didn’t ask for. Or deploy everything on your cloud provider without code review. It’s just trying to be helpful. You scold it, tell it to never do that again. And then several cycles later, it slips. Then you insist, never EVER do that again. Make it a memory. Burn it in your head. And then it slips again. And again.

The forgetting issue

If you happen to ask an agent some compaction rounds later what you said about X it won’t know and chances are it will make something up1. Why doesn’t a system centered on written text fail to recall that very same text? Not remember the whole conversation, although it certainly wouldn’t hurt to be able to do that as well, just the things that you, the user, the driver of the whole conversation, has said about something specific not that long ago. It does not even know what time it is now by default, you ’ll have to make it check, so if you continue a conversation from last month, it just assumes it’s the next day. Sure, there are issues with paraphrasing yourself or context window size, but storing text and searching through it seems to be relatively straightforward these days2,3.

The changing your mind issue

Over long running conversations, context gets lost. During compaction, text gets summarized. But we should be able to know what was said and in what order. If you change your mind later, if the goal changes mid conversation, if new knowledge supersedes old assumptions and you need to backtrack and make a different turn, we can find similarly worded prompts and replace them. If you first said X and then made it Y, we can resolve these conflicting statements over time by overwriting the older one4,5,6. The agent should have up to date context at every response or action about previously stated invariants or preferences. It might not always behave exactly as we want it to, but it’s a step in the right direction. Relevant information should be surfaced even without perfect retrieval reliability or even perfect relevance. The agent might choose to ignore that context, but it should be fed to it to, at the very least, be nudged in the right direction.

The importance issue

But even if we can get past up to date instructions into context have we solved the disobedience issues? Probably not. Relevance is not the same as recency or similarity based retrieval. Maybe the user really intended something else. But if it wasn’t stated we can’t get to it. We now have access to all things stated but which ones are pertinent? There’s been a lot of research on how to judge what’s important in context and how to create hierachical memories7. It has not led to a definitve solution, the problem seems to resist a consistent solution, no less because much is implied and not explicilty stated. But sometimes it is. Sometimes, the user just has to repeat something over and over and over again simply because the agent keeps ignoring them. Sometimes the user just makes it clear, in utter frustration and strong language that you are really not to do that ever again. It’s absurd to assume that’s not important, and ignore them half an hour later.

Context augmentation

Perfect solutions being out of reach, we can try to fix some of the bigger annoyances by implementing deterministic goading systems around disobedient LLM agents. Storing the user prompts both in a vector db for similarity and verbatim in a doc store to search should be trivial enough with current tech and not too expensive. We can paraphrase the user’s prompt and dig out even more relevant context. If the user asks ‘What did i say about X two weeks ago?’ we can paraphrase X, filter by thread and time range, retrieve both vectors and written text and add everything to the chatbot’s context. We should not rely on it to use the tools, call APIs or retrieve context if and when it decides to. We should strive not to feed it conflicting information, so if we detect changes on things the user has said, we should replace the older prompts with the updated ones in the RAG store. The similarity search will tell us if the user is repeating themselves, and we can detect strong language. We can use this to detect invariants, strong preferences and forbiden actions the agent has to avoid.

Still better than the status quo

Agents tend to ignore previous instructions and hardcoded memories, but if we feed it only the relevant...