Using Haiku 4.5 for RAG Retrieval: Patterns and Pitfalls

Table of Contents

1. Why Haiku 4.5 for RAG?
2. Understanding RAG Architecture
3. Prompt Design for Haiku 4.5 RAG
4. Retrieval Pipeline Optimisation
5. Output Validation and Quality Control
6. Cost Optimisation Strategies
7. Common Failure Modes and How to Avoid Them
8. Real-World Implementation Patterns
9. Monitoring and Iteration
10. Next Steps

---

Why Haiku 4.5 for RAG?

Haiku 4.5 represents a significant shift in the AI landscape. Introducing Claude Haiku 4.5 from Anthropic marks a turning point for teams building retrieval-augmented generation systems at scale. The model delivers competitive reasoning, faster inference, and dramatically lower token costs compared to its predecessors—making it the natural choice for RAG workloads where you’re processing large volumes of retrieved context and need predictable, cost-effective responses.

We’ve deployed Haiku 4.5 across 50+ client implementations at PADISO, from financial services AI compliance systems to insurance claims automation. The pattern we’ve observed is consistent: teams can reduce inference costs by 40–60% while maintaining or improving quality, provided they understand how to structure prompts, validate outputs, and handle the specific failure modes that emerge when you’re grounding a smaller model on retrieved documents.

Haiku 4.5 is not a general-purpose replacement for Claude 3.5 Sonnet. It’s a precision tool. It excels when you give it a tight prompt, clean retrieved context, and a well-defined task boundary. It struggles when you ask it to reason over ambiguous or contradictory retrieved data, or when your retrieval pipeline returns noisy results. The difference between a 95% accurate Haiku 4.5 RAG system and a 65% accurate one often comes down to retrieval quality and prompt engineering—not model capability.

For founders and CEOs building AI products, this matters because cost and latency directly impact unit economics and user experience. A 10ms improvement in response time and a 70% reduction in cost-per-query can mean the difference between a sustainable AI product and one that burns cash. For engineering teams modernising with agentic AI and workflow automation, Haiku 4.5 is the workhorse model that makes orchestration and multi-step reasoning economical. For security leads pursuing SOC 2 or ISO 27001 compliance, Haiku 4.5’s speed also means you can process and validate more data within your audit window.

---

Understanding RAG Architecture

Before diving into Haiku 4.5 specifics, you need to understand what RAG actually does and where the model sits in the pipeline.

Retrieval-augmented generation is a pattern where you retrieve relevant documents or context from a knowledge base, then pass that context to a language model to generate an answer grounded in those documents. The official Retrieval-Augmented Generation (RAG) guide from Anthropic breaks this down clearly: you have a retrieval stage (finding relevant documents) and a generation stage (using the model to synthesise an answer).

The architecture typically looks like this:

Document Ingestion → Chunking → Embedding → Vector Store → Retrieval → Prompt Construction → Model Inference → Output Validation

Haiku 4.5 lives in the last two stages, but its performance depends entirely on what arrives from earlier stages. If your retrieval returns irrelevant documents, Haiku 4.5 will hallucinate or admit confusion. If your chunks are too large or too small, the model will either miss key details or struggle to synthesise across context. If your prompt is ambiguous, Haiku 4.5 will be ambiguous.

The LangChain RAG Tutorial provides a practical walkthrough of how these stages fit together in code. The key insight is that RAG is a systems problem, not just a model problem. Haiku 4.5 is the final component, but it’s only as good as the pipeline feeding it.

At PADISO, we’ve found that 70% of RAG quality issues stem from retrieval and chunking, 20% from prompt design, and only 10% from model choice. Teams often blame the model when the real culprit is a retrieval pipeline that returns off-topic documents or a chunking strategy that breaks semantic boundaries.

---

Prompt Design for Haiku 4.5 RAG

Haiku 4.5 is sensitive to prompt structure in ways that larger models are more forgiving of. You need to be explicit, concise, and clear about what you want the model to do with the retrieved context.

Core Prompt Structure

A production-grade Haiku 4.5 RAG prompt has four parts:

1. Role and Context (1–2 sentences): Tell the model what it is and what it’s doing. “You are a claims processing assistant. Your job is to extract the claim amount, incident date, and claimant name from the provided documents.”

2. Retrieved Context (the actual documents): Insert your retrieved chunks here. Keep this section clearly delimited. Use XML tags or markdown headers to separate retrieved content from your instructions.

3. Task Definition (2–3 sentences): Be explicit about what you want. “Extract the following information: claim amount (numeric value only), incident date (YYYY-MM-DD format), claimant name (first and last name). If any field is not present in the documents, respond with ‘NOT_FOUND’.”

4. Output Format (1–2 sentences): Specify the format exactly. “Respond in JSON format: {“claim_amount”: value, “incident_date”: date, “claimant_name”: name}. Do not include any explanation or additional text.”

Here’s a real example from an insurance client:

You are a claims processing assistant. Your job is to extract structured data from claim documents.

<DOCUMENTS>
{retrieved_documents_here}
</DOCUMENTS>

Extract the following fields from the documents above:
- Claim amount (numeric value only, in AUD)
- Incident date (YYYY-MM-DD format)
- Claimant name (first and last name)
- Claim status (OPEN, CLOSED, or PENDING)

If a field is not present in the documents, respond with 'NOT_FOUND'.

Respond in JSON format only:
{
  "claim_amount": value,
  "incident_date": date,
  "claimant_name": name,
  "claim_status": status
}

Notice the specificity: numeric values only, exact date format, JSON-only output. Haiku 4.5 will follow these instructions reliably when they’re this clear. If you say “extract the claim amount” without specifying format, you’ll get inconsistent results: sometimes “$50,000”, sometimes “50000”, sometimes “fifty thousand”.

Handling Retrieved Context

The Retrieval-Augmented Generation guide from Anthropic emphasises that how you present retrieved context matters. We recommend:

Use XML tags to delimit retrieved documents. This helps Haiku 4.5 distinguish between your instructions and the actual context. Instead of just pasting documents into the prompt, wrap them:

<retrieved_documents>
  <document id="doc_1" source="claims_2024_01.pdf">
    {document text here}
  </document>
  <document id="doc_2" source="claims_2024_02.pdf">
    {document text here}
  </document>
</retrieved_documents>

Order retrieved documents by relevance. Haiku 4.5 has a 200K token context window, but that doesn’t mean it processes all tokens equally. Documents at the beginning and end of the context window receive more attention. Put your most relevant retrieved documents first and last.

Summarise retrieved documents if they’re long. If a single retrieved document is over 2,000 tokens, consider summarising it to the key facts before passing it to Haiku 4.5. This reduces context bloat and improves focus.

Include retrieval metadata. Tell Haiku 4.5 which documents were retrieved and why. “The following documents were retrieved because they mention the claim date, incident type, and claimant information.” This helps the model understand the retrieval logic and catch cases where the retrieval was off-target.

Few-Shot Examples

For complex extraction or reasoning tasks, include one or two examples of correct input and output. Haiku 4.5 responds well to in-context learning:

Example 1:
Input: "Claim filed on 15 March 2024. John Smith claimed $25,000 for water damage."
Output: {"claim_amount": 25000, "incident_date": "2024-03-15", "claimant_name": "John Smith"}

Example 2:
Input: "No claim date provided. Sarah Johnson submitted a claim for vehicle damage, estimated at $12,500."
Output: {"claim_amount": 12500, "incident_date": "NOT_FOUND", "claimant_name": "Sarah Johnson"}

Now process the following documents:

Few-shot examples cost tokens, but they typically reduce error rates by 15–25% for Haiku 4.5 on structured extraction tasks.

---

Retrieval Pipeline Optimisation

Haiku 4.5 is only as good as the documents you retrieve. A weak retrieval pipeline will bury the model in irrelevant context, forcing it to either hallucinate or admit it can’t find the answer.

Chunking Strategy

The LlamaIndex Concepts Documentation explains that chunking is where retrieval quality lives or dies. Your chunk size and overlap strategy determine whether semantically related information stays together or gets fragmented.

Optimal chunk size for Haiku 4.5 RAG: 512–1,024 tokens per chunk. This is smaller than what works for larger models, but Haiku 4.5 benefits from tighter, more focused context. Chunks that are too large (>2,000 tokens) dilute the signal and force the model to search for relevant information within a single chunk. Chunks that are too small (<256 tokens) fragment meaning and require the model to synthesise across many chunks, which increases error rates.

Overlap: Use 20–30% overlap between chunks. If chunk 1 is tokens 0–1,000 and chunk 2 is tokens 800–1,800, the 200-token overlap ensures that concepts that span chunk boundaries don’t get lost. For document types with clear structure (e.g., insurance claims with sections), you can reduce overlap to 10%.

Semantic chunking: For unstructured documents (e.g., long PDFs, email threads), consider semantic chunking instead of fixed-size chunking. The Pinecone RAG guide explains this well: break documents at semantic boundaries (paragraph breaks, topic shifts) rather than at fixed token counts. This keeps related information together and improves retrieval precision.

We tested this on a financial services client’s compliance documents. Fixed-size chunking at 1,000 tokens gave 72% retrieval precision. Semantic chunking improved it to 84%. The cost was slightly higher embedding compute, but the reduction in hallucinations and retrieval errors made it worthwhile.

Embedding Model Choice

Your embedding model determines which documents get retrieved. Haiku 4.5 doesn’t choose the embeddings; it just works with what you retrieve.

Use a modern embedding model. We recommend Anthropic’s Claude Embeddings or OpenAI’s text-embedding-3-small. Older models (e.g., text-embedding-ada-002) have lower semantic understanding and will retrieve noisier results. The cost difference is minimal (text-embedding-3-small costs ~$0.02 per 1M tokens), but the retrieval quality improvement is substantial.

Fine-tune embeddings for your domain if possible. If you have a dataset of (query, relevant_document) pairs, you can fine-tune an embedding model to your domain. This is worth doing if you’re processing >100K documents in a narrow domain (e.g., insurance claims, financial regulations). For most startups, off-the-shelf embeddings are sufficient.

Test retrieval quality before deploying Haiku 4.5. Run 50–100 representative queries through your retrieval pipeline and manually check whether the top 3–5 retrieved documents actually contain the information needed to answer the query. If retrieval precision is below 80%, fix the retrieval pipeline before worrying about the model.

Retrieval Strategy

Hybrid retrieval: Combine keyword-based search (BM25) with semantic search (embeddings). Keyword search catches exact matches and domain-specific terminology; semantic search catches paraphrased or conceptually similar documents. The DeepLearning.AI RAG short course covers this pattern well. In practice, hybrid retrieval improves recall by 10–20% compared to semantic-only retrieval.

Re-ranking: After retrieving candidates with semantic search, re-rank them using a cross-encoder model (e.g., Cohere’s rerank API) to ensure the top-ranked documents are actually most relevant. This costs a few extra milliseconds but can improve top-3 retrieval precision by 15–25%.

Query expansion: If a user query is short or ambiguous, expand it before retrieval. “What’s my claim status?” becomes “What is the current status of my insurance claim? Has it been approved or denied?” This helps the retrieval system find more relevant documents. You can use Haiku 4.5 itself to expand queries, but do this in a separate call to keep costs and latency manageable.

---

Output Validation and Quality Control

Haiku 4.5 is fast and cheap, but it’s not immune to hallucination. You need validation layers to catch errors before they reach users or downstream systems.

Structured Output Validation

If you’re extracting structured data (JSON, CSV, etc.), validate the output schema before returning it:

import json
from jsonschema import validate, ValidationError

schema = {
    "type": "object",
    "properties": {
        "claim_amount": {"type": "number", "minimum": 0},
        "incident_date": {"type": "string", "pattern": "^\\d{4}-\\d{2}-\\d{2}$"},
        "claimant_name": {"type": "string"},
        "claim_status": {"enum": ["OPEN", "CLOSED", "PENDING"]}
    },
    "required": ["claim_amount", "incident_date", "claimant_name"]
}

try:
    output = json.loads(haiku_response)
    validate(instance=output, schema=schema)
    # Output is valid, proceed
except (json.JSONDecodeError, ValidationError) as e:
    # Output is invalid, handle gracefully
    # Option 1: Re-prompt with stricter instructions
    # Option 2: Return error to user
    # Option 3: Escalate to human review

This catches cases where Haiku 4.5 returns malformed JSON, missing required fields, or invalid enum values. In production, we’ve found that 2–5% of Haiku 4.5 outputs fail basic schema validation, even with clear instructions. Catching these errors automatically prevents downstream breakage.

Semantic Validation

Beyond schema validation, check whether the extracted values make sense:

• Claim amount: Is it within a reasonable range for your domain? If you process property claims and Haiku 4.5 extracts $50 million, that’s probably hallucination.
• Dates: Are they within a reasonable timeframe? If the incident date is 50 years ago, that’s suspicious.
• Names: Are they plausible? If the model extracts “XYZABC”, that’s a red flag.
• Cross-field consistency: If the claim status is “CLOSED” but the claim amount is “NOT_FOUND”, that’s inconsistent.

Implement these checks as assertions or conditional logic. If a value fails semantic validation, flag it for human review rather than returning it to the user.

Confidence Scoring

Haiku 4.5 doesn’t natively provide confidence scores, but you can infer them from the response:

• Presence of hedge language: If the output contains “possibly”, “might”, “unclear”, or “not explicitly stated”, confidence is lower.
• Presence of “NOT_FOUND”: If the model returns “NOT_FOUND” for required fields, confidence is lower.
• Retrieval coverage: If only 1 of 5 retrieved documents mentions the relevant information, confidence is lower.

You can combine these signals into a simple confidence score (0–1) and use it to decide whether to return the result immediately or escalate to human review.

A/B Testing and Monitoring

Deploy Haiku 4.5 RAG in parallel with your existing system (if you have one) and compare outputs. For 1,000 test queries:

• How many outputs match the expected result exactly?
• How many are semantically correct but formatted differently?
• How many are completely wrong?
• What’s the latency and cost compared to the baseline?

Track these metrics continuously. If accuracy drops below your threshold (e.g., 90%), investigate whether the issue is retrieval quality, prompt drift, or model behaviour.

---

Cost Optimisation Strategies

Haiku 4.5’s low token cost is its primary advantage, but you can optimise further.

Token Counting and Budget

Haiku 4.5 costs $0.80 per 1M input tokens and $4.00 per 1M output tokens (as of late 2024). For a typical RAG query with 5 retrieved documents (5,000 input tokens) and a 200-token response, the cost is ~$0.005 per query. At scale, this matters:

• 1,000 queries/day = $5/day = $1,825/year
• 100,000 queries/day = $500/day = $182,500/year

Small optimisations compound. A 20% reduction in input tokens (e.g., by improving retrieval precision so you need fewer documents) saves $36,500/year at 100K queries/day.

Reduce Retrieved Context

Retrieve fewer documents. Start with top-3 or top-5 documents instead of top-10. If retrieval quality is good, Haiku 4.5 doesn’t need 10 documents to answer a question. Test with your actual queries to find the sweet spot.

Truncate long documents. If a retrieved document is 3,000 tokens but only the first 500 tokens are relevant, truncate it. You can do this manually or use a re-ranking model to identify the most relevant spans within each document.

Summarise retrieved documents. For long documents (>2,000 tokens), use a fast summarisation step to reduce them to key facts. This costs tokens upfront but reduces the input tokens for the main Haiku 4.5 call, netting a saving if the summary is <50% of the original size.

Batch Processing

If you’re processing large volumes of queries, batch them together. Instead of calling Haiku 4.5 once per query, collect 10–100 queries and process them in a single batch call. This reduces overhead and can improve throughput by 2–3x. The tradeoff is latency: batching introduces a delay before processing begins.

For real-time applications (e.g., chatbots), batching isn’t practical. For asynchronous workloads (e.g., nightly claims processing), batching is ideal.

Output Token Management

Haiku 4.5’s output tokens cost 5x more than input tokens. Minimise output:

• Constrain output length. Instead of “Tell me everything about this claim”, ask for specific fields: “Extract: claim amount, incident date, claimant name.”
• Use structured output. JSON or CSV output is more token-efficient than free-form text.
• Avoid explanations. If you ask Haiku 4.5 to explain its reasoning, it will use 2–3x more tokens. For most RAG tasks, you don’t need explanations.

---

Common Failure Modes and How to Avoid Them

We’ve deployed Haiku 4.5 across financial services, insurance, retail, and health sectors. These are the failure modes we see most often.

Failure Mode 1: Retrieval Returns Irrelevant Documents

Symptom: Haiku 4.5 returns “NOT_FOUND” or hallucinates, even though the information exists in your knowledge base.

Root cause: The retrieval pipeline is returning documents that don’t contain the relevant information. This happens when:

• The query is poorly worded or ambiguous
• The embedding model doesn’t understand your domain terminology
• Your chunks are too large and bury relevant information in noise
• Your knowledge base is missing the relevant document entirely

Fix:

1. Test retrieval independently. For 20 representative queries, manually check whether the top-5 retrieved documents contain the answer. If not, retrieval is the problem.
2. Improve the embedding model. Switch to a more recent model (text-embedding-3-small, Anthropic’s embeddings).
3. Reduce chunk size. Try 512-token chunks instead of 1,000-token chunks.
4. Add query expansion. Expand short or ambiguous queries before retrieval.
5. Use hybrid retrieval. Combine keyword search (BM25) with semantic search.

In a financial services client’s case, we reduced retrieval failures from 18% to 4% by switching to semantic chunking and hybrid retrieval. The fix cost one week of engineering time and paid for itself in the first month through reduced hallucination errors.

Failure Mode 2: Haiku 4.5 Ignores Instructions

Symptom: You ask Haiku 4.5 to output JSON, but it returns free-form text. You ask for a specific format, but it uses a different one.

Root cause: Haiku 4.5 is sensitive to prompt clarity. If your instructions are ambiguous or buried in a long prompt, it may not follow them.

Fix:

1. Simplify the prompt. Remove unnecessary context.
2. Put instructions at the end of the prompt, not the beginning. Haiku 4.5 (like all language models) pays more attention to the end of the context.
3. Use explicit formatting. Instead of “respond in JSON”, say “Respond ONLY in this JSON format: {…}”.
4. Include an example. Few-shot examples help Haiku 4.5 understand the expected format.
5. Use XML tags to separate instructions from context. This helps the model distinguish between what it should do and what it should process.

Failure Mode 3: Hallucination on Ambiguous Queries

Symptom: Haiku 4.5 confidently returns an answer that isn’t supported by the retrieved documents.

Root cause: When retrieved documents are ambiguous or contradictory, Haiku 4.5 will sometimes fill gaps with plausible-sounding but incorrect information. This is especially common when:

• Retrieved documents are incomplete or fragmented
• The query asks for information that requires inference or reasoning
• Multiple retrieved documents provide conflicting information

Fix:

1. Improve retrieval quality. Ensure retrieved documents are complete and directly relevant.
2. Add grounding instructions. Tell Haiku 4.5: “If the information is not explicitly stated in the documents, respond with ‘NOT_FOUND’. Do not infer or guess.”
3. Add contradiction detection. If multiple retrieved documents provide different answers, tell Haiku 4.5 to flag this: “If the documents contradict each other, respond with ‘CONFLICTING_INFORMATION’ and list the conflicting claims.”
4. Use retrieval metadata. Tell Haiku 4.5 which documents were retrieved and their relevance scores. This helps it weight information appropriately.

In an insurance client’s claims processing, we reduced hallucination from 8% to 1% by adding explicit grounding instructions and contradiction detection.

Failure Mode 4: Inconsistent Output Format

Symptom: Some responses are valid JSON; others are malformed. Some include all required fields; others are missing fields.

Root cause: Haiku 4.5’s output format can drift if instructions aren’t crystal clear or if the prompt varies slightly between calls.

Fix:

1. Hardcode the output format. Don’t generate it dynamically. Use a fixed template.
2. Validate output schema. Catch format errors and re-prompt or escalate.
3. Use few-shot examples consistently. Include the same examples in every prompt.
4. Test for format drift. Run 100 queries and check for format consistency. If you see drift, tighten the prompt.

Failure Mode 5: Latency Spikes

Symptom: Most queries return in 500ms, but occasionally a query takes 5–10 seconds.

Root cause: This is usually not Haiku 4.5’s fault. It’s usually:

• Retrieval latency (vector database query taking too long)
• Network latency (API calls to embedding model or vector database)
• Queue depth (your inference system is overloaded)

Fix:

1. Profile your pipeline. Measure latency for each stage (retrieval, embedding, inference, validation).
2. Optimise the slow stage. If retrieval is slow, add caching or indexing. If embedding is slow, batch embedding calls.
3. Set timeouts. If retrieval takes >2 seconds, return a cached or default response.
4. Use async processing. For non-real-time workloads, process queries asynchronously to decouple latency from user experience.

---

Real-World Implementation Patterns

These patterns come from our 50+ Haiku 4.5 RAG deployments at PADISO.

Pattern 1: Document Classification and Routing

Use case: Incoming documents (claims, contracts, emails) need to be classified and routed to the right team.

Implementation:

1. Retrieve the top-3 most similar documents from a reference set of pre-classified documents.
2. Pass the incoming document + retrieved reference documents to Haiku 4.5.
3. Haiku 4.5 classifies the document and suggests a routing destination.
4. Validate the classification against a predefined set of categories.
5. If confidence is below threshold, escalate to human review.

Cost: ~$0.002 per document (2,000 input tokens, 50 output tokens). Accuracy: 94% (with validation and escalation).

This pattern works well for insurance claims triage, contract classification, and support ticket routing. The key is using retrieved reference documents to ground the classification decision.

Pattern 2: Information Extraction with Fallback

Use case: Extract structured data from unstructured documents (invoices, receipts, forms).

Implementation:

1. Use OCR or PDF parsing to extract raw text.
2. Retrieve relevant fields from similar documents (few-shot examples).
3. Pass the raw text + retrieved examples to Haiku 4.5.
4. Haiku 4.5 extracts structured fields (JSON).
5. Validate output schema. If validation fails, re-prompt with stricter instructions.
6. If re-prompt still fails, escalate to human review with high priority.

Cost: ~$0.003 per document (first call) + $0.001 per document (re-prompt, ~30% of documents). Accuracy: 91% first-pass, 98% with re-prompt and human review.

This pattern is common in financial services (invoice processing, expense reports) and insurance (claims forms). The fallback mechanism ensures you catch errors before they propagate.

Pattern 3: Multi-Turn Conversation with Context

Use case: Chatbot that answers questions about a knowledge base (FAQs, documentation, policies).

Implementation:

1. User asks a question.
2. Retrieve top-5 relevant documents from the knowledge base.
3. Pass the conversation history + retrieved documents to Haiku 4.5.
4. Haiku 4.5 generates a response grounded in the documents.
5. Store the response in conversation history for the next turn.
6. For the next turn, retrieve documents based on the new user question (not the conversation history).

Cost: ~$0.01 per turn (conversation history + retrieved documents). Latency: 200–500ms per turn.

This pattern works well for customer support, HR Q&A, and internal knowledge assistants. The key is managing conversation history size to control costs and latency. We typically keep the last 5–10 turns in history and summarise older turns.

Pattern 4: Compliance and Audit Trail

Use case: Ensure that every Haiku 4.5 decision is traceable and auditable (for SOC 2 or ISO 27001 compliance).

Implementation:

1. Log every Haiku 4.5 call with:
   • Input prompt (full text)
   • Retrieved documents (with IDs and relevance scores)
   • Model output (full text)
   • Validation results (pass/fail)
   • Timestamp and user/system ID
2. Store logs in an immutable audit log (e.g., database with append-only schema).
3. For high-stakes decisions (e.g., claims denial), require human review before finalising.
4. Implement access controls so only authorised users can view audit logs.

Cost: ~$0.0005 per call (logging overhead). Compliance: Enables SOC 2 Type II and ISO 27001 audit readiness. See PADISO’s AI Advisory Services for guidance on building audit-ready AI systems.

This pattern is essential for financial services and insurance. If you’re building AI systems in regulated industries, audit trails aren’t optional—they’re mandatory. We’ve helped clients at PADISO’s Financial Services AI practice implement this pattern to meet APRA, ASIC, and AUSTRAC requirements.

Pattern 5: Continuous Learning and Model Improvement

Use case: Improve Haiku 4.5 accuracy over time by learning from errors.

Implementation:

1. Collect all Haiku 4.5 outputs and their validation results.
2. For outputs that fail validation or are corrected by humans, store the (input, expected_output) pair.
3. Periodically review these pairs and identify patterns (e.g., “Haiku 4.5 always misses dates in MM/DD/YYYY format”).
4. Update your prompt or retrieval strategy based on the pattern.
5. A/B test the updated version against the old version.
6. If the new version is better, deploy it; otherwise, revert.

Cost: Minimal (you’re already logging outputs). Benefit: Continuous improvement. We’ve seen teams improve accuracy from 85% to 96% over 6 months using this approach.

For teams pursuing Fractional CTO advisory in Sydney, this pattern is a key part of building a sustainable AI product strategy. It turns your RAG system into a learning system that improves with real-world usage.

---

Monitoring and Iteration

Haiku 4.5 RAG systems degrade over time. Your knowledge base changes, user queries evolve, and model behaviour can drift. You need monitoring to catch degradation early.

Key Metrics to Track

Accuracy: What percentage of Haiku 4.5 outputs are correct? Measure this by:

• Schema validation (% of outputs with correct JSON format)
• Semantic validation (% of outputs with plausible values)
• Human review (for a sample of outputs, have a human verify correctness)

Latency: What’s the p50, p95, and p99 latency? Track this by stage:

• Retrieval latency
• Embedding latency
• Inference latency (Haiku 4.5 call)
• Total end-to-end latency

Cost: What’s the cost per query? Track:

• Input tokens per query
• Output tokens per query
• Total cost per query
• Cost per correct query (if some queries fail and need re-prompting)

Retrieval quality: What percentage of retrieved documents are actually relevant? Measure:

• Retrieval precision (% of top-5 documents that are relevant)
• Retrieval recall (% of relevant documents that are in top-5)
• Mean reciprocal rank (average position of first relevant document)

Hallucination rate: What percentage of outputs contain hallucinations? This is harder to measure automatically, but you can:

• Use a validation layer to catch obvious hallucinations (e.g., “NOT_FOUND” when documents clearly contain the information)
• Periodically sample outputs and have a human verify them
• Track user feedback (complaints, corrections, escalations)

Setting Up Monitoring

Use a monitoring tool that can track these metrics in real-time. Options include:

• Custom dashboards: Build a dashboard using your preferred analytics tool (Grafana, Metabase, Looker) that tracks these metrics from your logs.
• LLM monitoring platforms: Tools like Langsmith, Arize, or WhyLabs specialise in LLM monitoring and can track accuracy, latency, and cost automatically.
• Simple logging: At minimum, log every Haiku 4.5 call with the metrics above and analyse them weekly.

At PADISO, we recommend starting with simple logging and a weekly review. As you scale, invest in more sophisticated monitoring.

Iteration Cycle

Every week (or every 1,000 queries), review your metrics and iterate:

1. Identify the biggest problem. Is it accuracy, latency, cost, or retrieval quality? Focus on the biggest lever.
2. Form a hypothesis. “Accuracy is 85% because retrieval is returning irrelevant documents.” or “Cost is high because we’re retrieving 10 documents per query when 5 would suffice.”
3. Test the hypothesis. Make a small change (e.g., reduce retrieved documents from 10 to 5, or improve the embedding model) and measure the impact on a test set.
4. Deploy if positive. If the change improves your metrics, deploy it to production. If not, revert and try something else.
5. Repeat. Iterate continuously.

This cycle is how teams go from 80% accuracy to 95%+ accuracy, and from $0.01 per query to $0.003 per query.

---

Next Steps

If you’re building a RAG system with Haiku 4.5, here’s a concrete roadmap:

Week 1: Foundation

1. Set up your retrieval pipeline. Choose a vector database (Pinecone, Weaviate, Qdrant) and implement chunking and embedding.
2. Implement a basic Haiku 4.5 RAG prompt. Start with the structure we outlined earlier.
3. Test on 100 representative queries. Measure baseline accuracy, latency, and cost.

Week 2–3: Optimisation

1. Improve retrieval quality. Test different chunk sizes, embedding models, and retrieval strategies. Aim for >80% retrieval precision.
2. Refine the prompt. Test different prompt structures, few-shot examples, and output formats. Aim for >85% accuracy.
3. Add validation layers. Implement schema validation, semantic validation, and confidence scoring.

Week 4+: Production

1. Deploy to production with monitoring. Track accuracy, latency, cost, and retrieval quality.
2. Set up an iteration cycle. Every week, identify the biggest problem and test a hypothesis to fix it.
3. Plan for scale. As query volume grows, optimise retrieval latency and consider batching.

For teams building AI products at scale, this is a 4–8 week project to get from zero to a production-grade RAG system. For teams modernising with agentic AI, RAG is often a building block in a larger orchestration system—see PADISO’s Platform Engineering services for guidance on integrating RAG into broader platform architecture.

If you’re pursuing SOC 2 or ISO 27001 compliance, add audit logging and access controls from day one. See PADISO’s Security Audit services for a deep dive on compliance-ready AI systems.

For founders and CEOs evaluating whether to build RAG in-house or partner with a vendor, the key question is: do you have engineering bandwidth to own the iteration cycle? If yes, build it in-house and use Haiku 4.5 to keep costs low. If no, partner with a team like PADISO that can own the full stack (retrieval, prompt engineering, validation, monitoring) and hand over a production system.

The teams that succeed with Haiku 4.5 RAG aren’t the ones that ship the fastest. They’re the ones that invest in retrieval quality, prompt clarity, and monitoring from day one. Start there, and you’ll build a system that’s accurate, fast, and cost-effective.