Prompt Caching: Test Warmth, Expiry and Real Savings

Why measure cache warmth instead of assuming it

Prompt caching can reduce repeated processing of a stable prompt prefix, but a cache hit is not automatically a saving. The useful question is whether a particular workload reuses the same eligible content often enough, within the provider’s actual expiry behavior, to justify the change. A longer-lived cache can still disappoint if requests arrive in isolated bursts, prompts vary by tenant, or the first request after expiry is expensive.

This tutorial builds a small, repeatable measurement exercise. You will record request characteristics, compare cache-enabled behavior with a baseline, test what happens around expiry, and check that caching does not change answer quality or expose one tenant’s information to another. The example is hypothetical; its request counts and arithmetic are illustrative, not an observed result or a price estimate.

The terms “five-minute” and “one-hour” are useful experimental windows only when the provider and configuration you use actually support those behaviors. Cache eligibility, time-to-live, and usage accounting are provider-specific; inspect the relevant provider guidance and measure the usage your integration actually reports rather than assuming universal TTLs or savings (provider documentation). The experiment below deliberately treats cache behavior as something to verify, not a promise to infer from configuration.

Prerequisites and setup

Before changing a production prompt path, identify one workload with a stable, repeated prefix and a clear business outcome. Suitable candidates might include a support assistant with shared instructions or a document workflow using a fixed extraction rubric. Do not begin with a prompt that embeds fresh customer records in what you believe is a static prefix. First map which content is shared, tenant-specific, user-specific, and request-specific.

You will need a staging or otherwise controlled environment, access to your application’s request and response records, a way to control request timing, and a representative set of test inputs. You also need a baseline path that sends requests without relying on a warm cache, if your provider and implementation allow a meaningful comparison. If the provider offers no switch for a clean baseline, use a separate controlled configuration or compare measured cold and warm requests while documenting the limitation.

Set up a small experiment log before sending traffic. At minimum, capture a pseudonymous request identifier, timestamp, prompt version, workload or tenant class, cache configuration label, input and output usage reported by the integration, latency, outcome status, and whether the response passed the quality and access checks. Add any provider-reported cache usage fields only after confirming their meaning for your specific integration. Do not treat field names copied from an example as universal API properties.

Keep request content out of general-purpose operational logs unless there is a justified, controlled need to retain it. For this exercise, prompt version hashes, synthetic tenant identifiers, and test-case IDs are usually enough to connect measurements without collecting the full prompt or customer payload. This is an experiment-design recommendation, not a claim that hashing alone makes sensitive data safe.

Prepare three prompt components in your application: a shared, stable instruction block; a tenant-specific policy or context block; and a changing request block. Give each a separately recorded version or hash. The design goal is to make the reuse boundary visible. If your system currently concatenates everything into one string, refactor the construction path so you can inspect the exact ordered components sent for each test.

Step 1: Define the experiment and success criteria

Write down what decision the test must support. For example: “Should we use the longer supported expiry profile for this stable workflow, retain the current profile, or leave caching disabled?” Avoid framing the goal as “prove caching works.” The experiment needs to be capable of finding no meaningful benefit, a quality regression, or an access-boundary defect.

Choose a primary measure and guardrails. A useful primary measure is the measured change in input-processing cost per successfully completed task, calculated from actual provider usage and your applicable contract rates. Guardrails can include response quality, tenant isolation, latency distribution, error rate, and prompt-version freshness. Set acceptable limits before viewing results so a favorable cost number does not quietly excuse a failed answer or a privacy defect.

Define the unit of comparison. Requests should be grouped by the same workload, prompt version, input class, and relevant tenant pattern. If the cached and baseline groups receive different tasks, request lengths, or prompt versions, the comparison will confound cache behavior with workload composition. A small balanced test set is more informative than a large pile of unmatched traffic.

Record the test window and the events that begin or end it. For a TTL experiment, include a sequence of requests immediately after a known cold start, a cluster inside the candidate window, and a request after a deliberate quiet interval that exceeds it. Do not assume the configured duration is the observed expiry boundary: the request trace must establish what happened.

Step 2: Separate stable content from changing content

Inspect the request assembly code and produce a component map. A simple map might label shared_instructions as global, tenant_policy as tenant-specific, case_context as request-specific, and user_question as dynamic. Check whether timestamps, experiment labels, randomized formatting, or changing retrieval results have been inserted into the stable portion. A cache experiment is uninterpretable if the supposedly identical prefix silently changes on every call.

Keep stable material together and put genuinely changing material after it where the integration and provider’s documented behavior support that arrangement. Do not move tenant-specific content into a global shared prefix merely to seek reuse. A high hit rate is not a successful result if it is achieved by combining data across trust boundaries or by using an obsolete policy.

For each test request, record hashes of the assembled components and the overall prompt version. A changed hash tells you that inputs differed; it does not tell you whether a cache hit occurred. That distinction matters: prompt identity is a condition you can inspect in your application, while cache behavior must be inferred from provider-supported signals or controlled observations.

A compact request trace

The following Python is an illustrative instrumentation pattern, not a provider SDK example. Adapt the field names to your own application and avoid storing raw prompts in the event log. The result object is deliberately treated as an adapter input: your integration must populate its usage fields from verified response data.

from dataclasses import dataclass
from datetime import datetime, timezone
import hashlib
import json

@dataclass
class RequestParts:
    shared: str
    tenant: str
    dynamic: str
    prompt_version: str
    tenant_test_id: str
    case_id: str


def fingerprint(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]


def build_trace(parts: RequestParts, result: dict) -> dict:
    """Illustrative adapter: map only verified integration fields."""
    return {
        "observed_at": datetime.now(timezone.utc).isoformat(),
        "prompt_version": parts.prompt_version,
        "tenant_test_id": parts.tenant_test_id,
        "case_id": parts.case_id,
        "shared_hash": fingerprint(parts.shared),
        "tenant_hash": fingerprint(parts.tenant),
        "dynamic_hash": fingerprint(parts.dynamic),
        "cache_profile": result.get("cache_profile_label"),
        "input_usage": result.get("verified_input_usage"),
        "cache_usage": result.get("verified_cache_usage"),
        "latency_ms": result.get("latency_ms"),
        "quality_pass": result.get("quality_pass"),
        "access_pass": result.get("access_pass"),
        "status": result.get("status"),
    }


def write_trace(event: dict) -> None:
    print(json.dumps(event, sort_keys=True))

Expected output is one structured event per request, with stable component hashes across requests intended to share a prefix and changing hashes where input content differs. A null cache-usage field means your adapter has not supplied a verified measure; it is not evidence of a hit or a miss. Fix the adapter or use a different measurement method before calculating savings.

Step 3: Establish a cold and warm request sequence

Run a short sequence with a fixed prompt version and fixed test payload shape. First send one request after a sufficiently long idle period or another documented cold-start condition. Then send repeated requests with identical stable components at known intervals. Finally, stop sending requests for a deliberate period and send another request. Record every timestamp rather than relying on a worker’s memory or a dashboard’s aggregate percentage.

A useful schedule could be: request A at minute 0, B at minute 2, C at minute 4, a pause, and D after the candidate short window; repeat the sequence with a longer pause chosen for the candidate extended window. Those timings are a proposed test design, not a claim that expiry occurs at an exact minute. If expiry timing is uncertain, add samples on both sides of the expected boundary and report the uncertainty.

Keep one variable changing at a time. In the first pass, hold the prompt version, tenant, and request shape constant while changing only elapsed time. In another pass, hold time close while changing one component deliberately: the tenant policy, a single instruction character, or the dynamic user content. This shows whether your own assembly logic preserves the intended stable prefix and whether the provider’s reported behavior responds as expected.

Capture cold-start and warm observations separately. Averages across the whole experiment can conceal a costly first request followed by many inexpensive ones, or a burst pattern that rarely reuses a prefix. For latency, retain individual observations or report suitable percentiles alongside the median; a single average is not enough to show the effect of occasional cold requests.

What a useful trace looks like

A simplified trace might show the same shared_hash and tenant_hash for requests A, B, and C, with different dynamic_hash values. The cache-usage observation could differ between A and the later requests, but only a verified provider signal or trustworthy usage record can support that interpretation. Request D after the pause helps identify whether a new creation-like event or other cold behavior appears.

If all three component hashes change, the test does not establish expiry. It establishes that request construction was not stable. If hashes remain constant but no cache-related signal is available, you can measure latency and total usage, but you cannot confidently attribute the difference to a cache hit. Mark that result inconclusive rather than inventing a hit rate.

Step 4: Run the controlled comparison

Use at least two groups: a baseline and the candidate cache profile. If testing a short and longer supported expiry, use three groups only when you can keep the workload and other settings comparable. Assign test cases to groups in a balanced way. Rotate request order when practical so that a temporary service slowdown does not affect only one configuration.

For each group, run enough repeated sequences to cover the normal arrival pattern. Include both bursts and quiet gaps if production traffic has both. If traffic is continuous during business hours but nearly absent overnight, a test made only of rapid requests will overstate likely reuse. Conversely, a test made only of isolated calls will understate reuse for a tightly clustered workflow.

Use the same task cases and scoring rubric across groups. For structured extraction, compare required fields, value accuracy, and an explicit “unknown” outcome when evidence is absent. For generated responses, have reviewers score correctness against a predetermined rubric without knowing which cache profile generated each response where practical. Record failures and retries; excluding them can make an expensive or unreliable setup look artificially efficient.

Do not infer monetary savings from input-token counts alone. Apply your actual contractual rates to verified usage categories, and include any costs that differ between the profiles. If those rates or cache-specific accounting cannot be confirmed, report usage deltas rather than dollars. A percentage reduction calculated with a guessed discount is not a measured financial result.

Step 5: Test answer accuracy and tenant access explicitly

Caching should not be accepted based only on a usage chart. Create a quality test set with expected answers, prohibited inferences, and cases where the correct response is to say that information is unavailable. Run it against the baseline and each candidate profile using the same prompt and payload versions. Compare both successful and failed outputs, including malformed responses and unsupported claims.

For tenant isolation, use synthetic records that contain unmistakably different canary values. For example, tenant north-test has a fictional account contact NORTH-CANARY-418, while south-test has SOUTH-CANARY-762. Ask each request to summarize only its own record, then deliberately interleave requests across tenants while keeping the shared system instructions constant. Search each response for the other tenant’s canary and verify that the correct tenant-specific answer is returned.

The expected result is not simply “no canary leak.” Confirm that the request builder attached the correct tenant context, that no test case was accidentally scored against the wrong tenant, and that a missing tenant context fails closed according to your application’s intended behavior. A cache hit or a cache miss cannot substitute for an application-level access check. Treat any cross-tenant canary appearance, wrong-tenant answer, or unexplained dependence on request order as a stop condition pending investigation.

This is particularly important in analytics products, where a shared prompt may coexist with tenant-specific data permissions. Keep caching design separate from row-level authorization; review the relevant production patterns for row-level security in Apache Superset if that is part of your stack. A prompt cache is not an access-control policy, and a passing synthetic test does not establish that every production data path is isolated.

Step 6: Calculate observed savings from your own trace

Consider this explicitly hypothetical workload: 1,200 requests in a day, a stable shared prefix recorded as 1,600 input tokens per request, and a variable portion of 700 tokens per request. Suppose a balanced experiment records 300 cold-like requests and 900 warm-like requests in one candidate profile. Assume, for illustration only, the applicable rate for the shared prefix is represented by R, and the variable portion’s rate by V. These symbols are not provider prices.

If the verified usage records show that the shared prefix is charged at the applicable full rate for 300 requests and at the applicable cached rate for 900 requests, the illustrative input cost is 300 × (1,600R + 700V) + 900 × (700V + 1,600C), where C is the measured applicable rate for cached shared-prefix usage. The baseline estimate for the same workload is 1,200 × (1,600R + 700V). The difference is a modeled input-cost delta, not a realized invoice saving unless all relevant billing categories and rates are confirmed.

This form makes assumptions visible. Replace R, V, and C with the rates actually applicable to your account and request category; use the integration’s verified usage quantities; then reconcile a sample period against billing records. If your accounting reports cached usage in a different way, adjust the equation rather than forcing the data into these categories. Never infer a universal cache discount from this example.

The break-even question is practical: does the measured difference exceed the cost and operational burden of maintaining the configuration, test coverage, and any added warm-up traffic? Include warm-up requests if you add them. A scheduled request that creates a cache entry may improve the next request’s experience, but it consumes resources and can be wasted if no user request follows. Test demand timing before making it routine.

Example measurement table

The figures below are invented solely to show how to organize results. They do not represent a PADISO system, a provider benchmark, or an expected outcome.

The apparent difference between 72 and 31 cold-like observations is useful only if the classification is based on verified usage data and the groups received comparable traffic. The 99% hash consistency also signals that a small portion of requests differed; investigate those records before attributing the entire result to expiry. Accuracy passes should be reviewed by error type, not treated as an interchangeable aggregate with access passes.

Step 7: Decide whether warmth, expiry, or savings justify a change

Use the trace to distinguish four outcomes. First, the cache may be warm often and reduce measured usage meaningfully. Second, the cache may be warm often but the financial effect may be small for your workload. Third, the longer window may have little effect because requests rarely repeat within it. Fourth, the result may be inconclusive because prompt identity or provider usage cannot be measured reliably. Each outcome implies a different next action; none should be relabeled as a success simply because a cache option was enabled.

Compare the observed idle gaps with the candidate expiry behavior. A longer window is most relevant when useful requests recur after the shorter profile has expired but before the longer profile would expire. If almost all repeated requests arrive within the short interval, extending expiry may add little. If the workload has long gaps, even a longer profile might not retain value. Segment by workload schedule rather than averaging unrelated traffic together.

Choose based on measured value and guardrails. Retain the baseline if the comparison is inconclusive or if accuracy and access checks fail. Keep the short profile if it captures most of the available reuse and the longer profile adds little. Consider the longer profile only when it is supported by the provider, the experiment shows additional useful reuse, and quality, isolation, and operational acceptance criteria pass. Do not pick a profile using request volume alone.

A proposed acceptance record can be short: prompt version tested; supported profile labels; test dates; number and shape of requests; cache measurement method; measured usage and latency; accuracy and access results; unexplained differences; decision owner; and rollback trigger. This makes the decision reproducible after the prompt changes or traffic mix shifts. It also prevents a one-time favorable test from becoming a permanent assumption.

A decision flow for the experiment

The flow below starts with the two conditions that make a caching comparison interpretable: stable request construction and a verified measurement method. It then separates a passing result from an experiment that must stop or be repaired. “Proceed” means continue with a measured candidate comparison, not deploy automatically.

flowchart TD
  accDescr: Workflow stages and decisions: Stable prompt components?, Repair request construction, Verified usage signal?, Improve measurement, Run expiry comparison, Accuracy and access pass?, Stop and investigate, Compare measured value. The adjacent text explains the conditions and exceptions.
  accTitle: Prompt Caching — Test Warmth, Expiry and Real Savings workflow
    A["Stable prompt components?"] -->|"No"| B["Repair request construction"]
    A -->|"Yes"| C["Verified usage signal?"]
    C -->|"No"| D["Improve measurement"]
    C -->|"Yes"| E["Run expiry comparison"]
    E --> F["Accuracy and access pass?"]
    F -->|"No"| G["Stop and investigate"]
    F -->|"Yes"| H["Compare measured value"]

Troubleshooting common results

No measurable cache activity appears. Check whether the prompt prefix is actually identical, whether the integration is recording the correct response information, and whether the tested content is eligible under the provider’s current rules. Confirm the profile configuration in the environment that produced the trace. If you cannot validate the signal, label the result inconclusive; do not manufacture a hit rate from stable hashes alone.

The first request after a pause does not look cold. Your pause may not exceed the tested expiry behavior, another process may have made a request, or the signal may not distinguish the event you expect. Record all traffic that could share the relevant prefix and run the test in a quiet, controlled environment. Treat a single ambiguous boundary observation as insufficient evidence.

Cache observations are lower than expected. Compare component hashes and inspect prompt assembly for timestamps, tenant identifiers, retrieval snippets, feature flags, whitespace normalization, and prompt-version changes. Then group results by the first differing component. This often reveals that a “shared” prefix is partly dynamic, though only your traces can establish the cause in your system.

The measured usage improves, but cost does not. Recheck the actual applicable rates and usage categories, then include the variable input, output, retries, and any warm-up traffic. Verify that the selected period is comparable to the baseline and reconcile a sample with billing information. If the result is still a small financial change, it may not warrant extra operational complexity.

Latency becomes less predictable. Break latency into individual cold-like and warm-like observations, then compare matching request classes and time periods. Check whether retries, queueing, request length, or upstream work changed between groups. Do not claim that a cache profile caused a latency shift when the trace cannot separate it from those other factors.

Accuracy changes in either direction. First verify that the prompt version and all task inputs match across comparison groups. Review individual disagreements against the rubric, particularly cases involving missing evidence, exact values, or tenant context. Caching is not a reason to lower the correctness bar; suspend the candidate if an unexplained quality regression remains.

A canary appears in the wrong tenant’s answer. Stop the test and preserve the minimal trace needed to diagnose request assembly and data flow. Confirm tenant identifiers, component hashes, concurrency behavior, and scoring labels. Do not resolve the finding by extending or shortening a TTL; isolate the access-boundary defect before resuming any cache experiment.

Printable decision worksheet

Use this compact worksheet in the experiment ticket or review record. It is a printable summary, not a claim that a downloadable template exists.

• Workload: Name the single workflow and describe its burst, idle-gap, and tenant patterns.
• Prompt boundary: List shared, tenant-specific, and request-specific components; record the versioning method for each.
• Provider behavior: Confirm which cache profiles are supported for the deployed integration and how their usage can be observed.
• Baseline: Capture the same test cases without relying on a warm cache, or document why a clean baseline is unavailable.
• Expiry sequence: Record cold-start conditions, request timestamps, repeated calls, quiet periods, and post-gap observations.
• Measurement: Store verified usage fields, applicable rates or usage-only deltas, latency observations, retries, and status.
• Accuracy: Score the same cases against a fixed rubric, including uncertainty and missing-information cases.
• Access: Run interleaved synthetic-tenant canary tests and verify that missing tenant context behaves as intended.
• Decision: State whether the result supports the baseline, short profile, longer profile, or no change; include evidence and uncertainty.
• Rollback trigger: Define the quality, access, or operational condition that returns the workload to its previous configuration.

Put the result into operating practice

Treat the measurement as tied to a prompt version and workload, not as a permanent property of your application. Re-run it when the stable instructions, tenant-context assembly, provider configuration, or traffic pattern changes materially. A small recurring sample can detect drift without turning every routine request into a formal experiment, provided the sample still records enough information to interpret the result.

Keep the change narrow when you move beyond staging. Route a limited, representative workload through the candidate configuration, retain a clear rollback path, and compare its outcomes with a contemporaneous baseline where feasible. Do not roll out solely because a configuration label says “longer.” The decision should still depend on actual usage, answer quality, access checks, and the operational cost of maintaining the behavior.

For teams building analytics experiences, the same discipline applies when AI-generated explanations sit beside tenant-scoped dashboards: measure the prompt reuse separately from the authorization and data-query path. If that broader implementation needs engineering support, PADISO’s embedded analytics service is a relevant next step. The cache experiment remains a bounded question: does this stable prompt recur often enough, under verified expiry behavior, to produce a worthwhile and safe measured result?

The practical outcome is not a universal winner between five minutes and one hour. It is a defensible choice for one workload, grounded in request traces, verified usage, explicit accuracy and access tests, and a calculation using your own applicable rates. If the test cannot support that choice, improve the experiment before changing the production default.

WANT TO TALK THROUGH YOUR SITUATION?

Talk it through with us. Thirty minutes, no slides.