wiki/tail-prompt-optimization-en.md at main · orbsh/wiki · GitHub
//blob/show" data-turbo-transient="true" />
Skip to content
Search or jump to...
Search code, repositories, users, issues, pull requests...
-->
Search
Clear
Search syntax tips
Provide feedback
--><br>We read every piece of feedback, and take your input very seriously.
Include my email address so I can be contacted
Cancel
Submit feedback
Saved searches
Use saved searches to filter your results more quickly
-->
Name
Query
To see all available qualifiers, see our documentation.
Cancel
Create saved search
Sign in
//blob/show;ref_cta:Sign up;ref_loc:header logged out"}"<br>Sign up
Appearance settings
Resetting focus
You signed in with another tab or window. Reload to refresh your session.<br>You signed out in another tab or window. Reload to refresh your session.<br>You switched accounts on another tab or window. Reload to refresh your session.
Dismiss alert
{{ message }}
orbsh
wiki
Public
Notifications<br>You must be signed in to change notification settings
Fork
Star
FilesExpand file tree
main
/tail-prompt-optimization-en.md
Copy path
Blame<br>More file actions
Blame<br>More file actions
Latest commit
History<br>History<br>History
123 lines (91 loc) · 5.53 KB
main
/tail-prompt-optimization-en.md
Copy path
Top
File metadata and controls<br>Preview
Code
Blame
123 lines (91 loc) · 5.53 KB
Raw<br>Copy raw file<br>Download raw file
OutlineEdit and raw actions
Cache Tree and Tail Prompt Optimization
1. Cache Tree (Cache Tree)
Multiple turns share a common KV cache prefix, forming a tree structure:
[system prompt + conversation history] ← trunk (shared cache)<br>/ \<br>[branch A: question A] [branch B: question B] [branch C: question C] ← branches (new computation)
Core properties:
Shared trunk : All branches reuse the parent's KV cache, no redundant computation
Independent branches : Each branch's new tokens are computed independently
Cache reuse : When switching branches, the common prefix cache remains available (within TTL)
Typical scenario: IM platform thread mechanisms. Each thread is a branch sharing the main thread's prefix cache. Usually, base information is pre-filled into cache first (e.g., loading project code/docs in coding scenarios), then different branches execute different tasks — each branch inherits the cached project context and only needs to compute its own task tokens.
Main thread (trunk)<br>├── thread_A: fix bug → inherits trunk cache + bug description<br>├── thread_B: add feature → inherits trunk cache + requirement description<br>└── thread_C: write tests → inherits trunk cache + test scope
Each thread's cost ≈ task description tokens (project context uses cache).
2. Tail Prompt Optimization
A temporary instruction injected at the end of a prompt, leveraging the KV cache bypass branch mechanism to perform specific tasks.
Tail prompts are a variant of cache trees — Cache Vine . One main trunk grows continuously, with leaves sprouting periodically; after leaves fall, the trunk continues growing.
[system prompt + conversation history] ─── trunk (accumulates continuously)<br>├── [tail prompt A + question] ← leaf (one turn, falls off)<br>trunk continues growing...<br>├── [tail prompt B + question] ← leaf (one turn, falls off)<br>trunk continues growing...<br>└── [tail prompt C + question] ← leaf (one turn, falls off)
Differences from cache trees:
Dimension<br>Cache Tree<br>Tail Prompt (Vine)
Trunk<br>Shared trunk<br>Same, but trunk accumulates continuously
Branch<br>Persistent thread<br>Temporary leaf (one turn)
Lifecycle<br>Cross-turn, user-driven<br>Single turn, system-injected
Controller<br>User selects branch<br>System decides when to sprout leaves
Typical use<br>Multi-topic parallelism<br>Background tasks (compression, extraction, audit)
KV Cache Utilization
KV cache (unchanged): [conversation history] ← trunk<br>bypass branch (new computation): [tail prompt + user question] ← leaf<br>→ LLM completes both in a single turn: answer user + execute tail prompt task<br>→ After turn ends, leaf falls off (tail prompt discarded), only results remain
Analogy to TCO (Tail Call Optimization): tail calls reuse the current stack frame; tail prompts reuse the current KV cache. Both are "tail" operations that reuse existing state.
Core Properties
Cache utilization : History portion uses KV cache; only the tail prompt is newly computed
Bypass branch : Does not modify the main trunk (conversation history), only appends temporary instructions at the end
One-shot : Removed from prompt after turn ends, not written to session
Control : Injector decides when and what to inject; Agent only executes
Compression Scenario: In-Place Replacement When Leaf Falls
Tail prompts use the vine pattern — one main trunk sprouts leaves, then continues growing. The compression scenario's special case — when the leaf falls, the preceding trunk is also replaced (checkpoint replaces old history).
KV cache (unchanged):<br>[ckpt_0] + [msg_101..msg_150]
Newly appended messages (only uncached...