How do I choose which prompt prefix to cache versus re-run to cut LLM latency?

A prompt-caching decision is not really about whether caching is good. It is about where the reusable boundary should sit when part of the request is stable and part of it changes on every turn. In a multi-part assistant prompt, the stable material is usually the system instructions, developer instructions, tool definitions, and other reference text that the model needs every time. The volatile material is the current task, the new user message, or the latest conversation turn that should not be reused blindly.

That distinction matters because OpenAI's prompt cache reuses the model's saved key-value state for an unchanged prefix. The cache does not store arbitrary tokens for later textual replay; it stores the internal state for the rendered prefix. If the beginning of the next request matches, the model can skip reprocessing those tokens. If the prefix changes before the breakpoint, reuse stops there, even if most of the later prompt looks similar.

The practical question, then, is not 'should I cache everything?' It is 'which prefix is stable enough to pay the cost of caching, and which suffix is too volatile to be worth keeping in that reusable region?' That is the question this article answers with a placement rule, a measurement rule, and a fallback rule.

The documented rule is prefix matching: a later request can reuse cache state only when it has the same rendered prefix and can find an eligible matching breakpoint. That rendered prefix includes OpenAI-provided instructions, developer messages, tool definitions, and conversation history, so changes anywhere in that early portion can invalidate reuse. The cache is therefore not just a feature you turn on once; it is a contract about which early tokens remain identical across requests.

The second rule is that each prefix you want to cache must meet the model's minimum cacheable length. That requirement prevents tiny prefixes from being treated as worthwhile cache entries. For a decision process, this means a prefix is only interesting if it is both stable and long enough to matter. A short, perfectly stable instruction block may still be too small to justify special treatment, especially if your application rarely repeats it.

The third rule is model and mode behavior. On GPT-5.6 and later, the docs distinguish implicit caching from explicit-only caching. Implicit caching places a breakpoint automatically at the end of the latest eligible message. Explicit-only mode lets you place breakpoints deliberately, which becomes important when the suffix changes often and you do not want it to incur cache-write cost. The same guide also documents cache TTL choices, including 30m and 12h minimum lifetimes, and notes that cache keys can be used to separate accounting or routing for customers, users, or workspaces.

The easiest way to pick a caching boundary is to classify each piece of the prompt by volatility. System instructions are usually stable. Tool descriptions and schemas are often stable until you change a tool contract. Conversation history is stable only up to the point where you append new turns. A current task prefix may be partly stable if you have a reusable rubric, but the live user request at the end is usually volatile.

The prompt in this article's example code makes that distinction explicit through a contiguous leading stable region. It marks the system message, developer instructions, and shared examples as stable, then stops at the first volatile user message. That is a useful model even when your real application is more complicated, because it mirrors how cache reuse works: only the unchanged beginning of the rendered prompt can be reused as one prefix. A stable block later in the prompt does not help unless the earlier content stays identical as well.

The decision should be based on content ownership, not on vibes. If a section changes when the user changes, it belongs after the breakpoint. If a section changes when you update a tool schema, it may belong in the cached region only if you are prepared for that update to invalidate the prefix. If a section is expensive to regenerate but rarely changes, it is a strong cache candidate. If a section changes every request, caching it is often a false economy because it converts a volatile suffix into a cache boundary that cannot be reused reliably.

A good caching decision starts with the size of the candidate prefix. The example helper in this article estimates token cost for each prompt part, totals the contiguous stable prefix, and compares that total with a minimum cacheable threshold. That is not the same as the provider's tokenizer, but it is enough to make the placement decision visible in a local fixture: you can see whether the stable block is likely to be large enough to matter before you build more elaborate instrumentation.

That measurement matters because the latency guide is clear that reducing input tokens usually has a much smaller effect than reducing output tokens, while prompt caching is about avoiding repeated processing of unchanged input. Those are related but different levers. If your stable prefix is tiny, the likely win from caching may be small. If your prefix is large and repeated across many calls, the same rule becomes much more attractive because the model can reuse the prefix state rather than reprocessing it each time.

The article's example also surfaces a practical boundary: a stable prefix below the minimum cacheable length should not be treated as cacheable in your decision flow, even if it feels logically reusable. That is an important negative case because it prevents teams from assuming that every reusable-looking block deserves its own cache plan. The right conclusion may be to shorten the prefix, merge adjacent reusable instructions, or move some instructions into a different layer of the application rather than trying to cache a block that is too short to qualify.

OpenAI's docs make two related controls available on GPT-5.6 and later: prompt cache mode and prompt cache key. Mode controls whether breakpoints are implicit or explicit-only. The key controls how cache accounting is separated across customers, users, workspaces, or other application scopes. That second control is not just a billing detail. It also helps keep repeated prefixes from different scopes from collapsing into one accounting bucket when your application needs to reason about reuse per tenant or per session.

This is where the article's example recommendation of separate keys for conversation history and volatile task parameters becomes useful. If you are processing a long-lived support thread, you may want the shared policy or rubric to remain on one stable key while per-user or per-session content gets its own key or its own uncached suffix. The point is not to invent artificial complexity; it is to keep the cache boundary aligned with the part of the prompt that should truly be shared. A different user, customer, or workspace should not inherit cache accounting just because the text looks similar.

The TTL choice also matters. The docs describe 30m and 12h minimum lifetimes, with 12h intended for stable instructions or reference material reused over longer gaps. The key point is that the TTL sets a minimum cache lifetime, not a guaranteed delete time. In practice, that means you choose a longer lifetime only when the prefix is likely to be reused enough to justify it. If the content changes frequently, a shorter lifetime can be the safer default. If the content is reference material or a hot prefix that many requests will reuse, the longer minimum lifetime is often a better fit.

A cache strategy needs a miss path. When the prefix does not match, or the request changes before the breakpoint, the model must still answer correctly at uncached input rates. The latency docs are useful here because they remind you that fewer requests, fewer output tokens, and smaller model work can all reduce user-visible delay even when caching does not help. Prompt caching should be one tool in the plan, not the only tool.

A practical fallback is to reduce what the model must reason about when reuse fails. If a cache miss occurs on a known hot prefix, you can shorten the prompt by trimming redundant instructions, removing unneeded history, or reducing output verbosity. The latency guide explicitly recommends maximizing the shared prompt prefix by moving dynamic material later in the prompt. If you cannot reuse the prefix, the next best improvement is to shrink the volatile work that remains.

That also explains two common gotchas from the docs. First, append-only updates can prevent reuse if they change the prefix before the breakpoint, because matching depends on the rendered prefix staying identical. Second, compaction can replace earlier conversation content with a compacted context, which may prevent reuse from the first changed token onward. Those are not bugs in the cache; they are reminders that prompt caching is sensitive to prompt shape. If you change the shape often, you should expect reuse to drop and design your application so that a miss still produces an acceptable latency budget and a correct answer.

The example implementation in this article is intentionally small: it finds the longest contiguous leading stable prefix, estimates its token cost, and proposes a cache mode based on whether the prefix is long enough and whether later stable parts make explicit-only safer. That is enough to support the article's decision flow, but it does not claim to simulate the provider's internal cache or to prove a real latency reduction by itself. It is a planning helper, not an execution harness.

That distinction is important because the outcomes you ultimately care about are measured on the real system: reduction in model prefix compute time, observed latency drop, and cache hit rate. The provider docs tell you what the cache does and what controls exist; your application metrics tell you whether the prefix you chose actually pays off. A good instrumentation plan counts the repeated prefix length, the rate of cache reuse, and the latency difference between cold and warm requests under the same user-visible workload.

The example also helps you see what the tests can and cannot prove. They can prove that a stable prefix above a threshold is classified as cacheable, that a short prefix is not, and that invalid input is rejected. They cannot prove provider-level reuse, retained cache duration beyond the documented minimum, or real model latency on a production network. The useful conclusion is therefore a decision rule: keep stable instructions and reference material in a reusable prefix, keep volatile task parameters and per-turn text outside that prefix, choose explicit-only when you want precise control over breakpoints, and verify the result with your own latency and hit-rate measurements.