Using Sonnet 4.6 for Real-Time Streaming Chat: Patterns and Pitfalls

Table of Contents

1. Why Sonnet 4.6 for Streaming Chat
2. Architecture Fundamentals
3. Prompt Design for Streaming
4. Output Validation and Safety
5. Cost Optimisation at Scale
6. Common Failure Modes
7. Real-World Implementation Patterns
8. Testing and Observability
9. Moving to Production

---

Why Sonnet 4.6 for Streaming Chat

Streaming chat applications demand a different set of trade-offs than batch inference. Your users expect token-by-token output within 100–200 milliseconds of the API request hitting your server. They’re watching a blinking cursor. Latency compounds across the entire request path: network round-trip, model inference time, token serialisation, and browser rendering. Every millisecond matters.

Claude Sonnet 4.6 is purpose-built for this workload. It balances speed, cost, and capability in a way that makes streaming chat economically viable at scale. Unlike larger models that introduce multi-second latencies, Sonnet 4.6 delivers the first token in under 500ms for most prompts, and subsequent tokens stream at rates that feel instantaneous to users.

The model also ships with a 200k token context window, which matters for chat applications where you need to retain conversation history, reference documents, or maintain system context without truncation. That context depth, combined with aggressive pricing ($3 per 1M input tokens, $15 per 1M output tokens), makes Sonnet 4.6 the pragmatic choice for production chat systems that need to handle thousands of concurrent users without burning through your infrastructure budget.

But speed and cost alone don’t guarantee a working system. Streaming introduces complexity. Your prompt design must account for token-by-token output, where the model can’t revise earlier tokens once they’ve been sent to the client. Your validation logic must operate on partial, incomplete responses. Your infrastructure must handle backpressure when clients disconnect mid-stream. And your cost model must account for the fact that streaming doesn’t reduce token consumption—it just distributes it over time.

This guide covers the patterns that work, the mistakes that cost money, and the failure modes you’ll hit if you rush to production without thinking through the mechanics.

---

Architecture Fundamentals

The Streaming Request Path

A real-time streaming chat request travels through your system in this order:

1. Client sends prompt – Browser or mobile app submits user input, conversation history, and any context (documents, user metadata).
2. Your backend queues the request – You validate input, check rate limits, and prepare the system message and prompt.
3. Your backend calls Anthropic’s streaming API – You open a persistent HTTP connection and begin receiving tokens.
4. Tokens stream to the client – Each token arrives as a separate event; your client-side code renders it into the DOM or UI buffer.
5. Stream completes or errors – The connection closes, and your backend logs the final token count and latency.

This path sounds simple, but the devil lives in the details. Each step has failure modes that will bite you in production.

HTTP/2 and HTTP/3 Considerations

Streaming performance depends heavily on your transport layer. HTTP/3, the latest standard, reduces head-of-line blocking and connection setup time. If your infrastructure supports it, HTTP/3 can shave 50–100ms off time-to-first-token. Most modern CDNs and cloud providers now support HTTP/3 natively, so it’s worth enabling on your reverse proxy or load balancer.

HTTP/2 is acceptable but introduces head-of-line blocking at the TCP layer. If you’re still on HTTP/1.1, upgrade immediately—streaming over HTTP/1.1 forces sequential token delivery and introduces unnecessary latency.

The Anthropic API supports streaming via server-sent events (SSE), a simple and well-supported pattern for pushing data from server to client. SSE is built on top of HTTP and requires no special client libraries or WebSocket complexity. For most chat applications, SSE is the right choice.

Concurrency and Connection Pooling

Your backend must maintain a pool of connections to Anthropic’s API. Each concurrent user stream requires one active connection. At scale—say, 1,000 concurrent users—you need robust connection pooling and backpressure handling.

Most HTTP clients (Python’s httpx, Node’s node-fetch, Go’s net/http) handle connection pooling automatically, but you need to configure pool size and timeout behaviour. A common mistake is setting connection timeouts too aggressively. Streaming requests can live for 10–30 seconds while tokens arrive. If your connection timeout is 5 seconds, you’ll drop streams mid-response.

Rule of thumb: set connection timeouts to at least 60 seconds, and use read timeouts (time between tokens) of 30 seconds. If no token arrives for 30 seconds, the connection has likely stalled, and you should close it and retry.

---

Prompt Design for Streaming

System Messages and Context Injection

Your system message sets the tone for the entire conversation. For streaming chat, keep it concise and explicit about output format. Verbose system messages don’t improve output quality—they just consume tokens and add latency.

A good system message for a streaming chat assistant looks like this:

You are a helpful assistant. Answer questions clearly and concisely.
If you don't know the answer, say so. Do not make up information.
Respond in the same language as the user's question.

That’s it. Anything longer is overhead. If you need to inject context (user metadata, conversation rules, domain-specific knowledge), do it in the user message, not the system message. The user message is where the actual work happens.

For domain-specific chat (customer support, technical assistance, etc.), inject context as structured data:

Context:
- User: John Smith
- Account status: Premium
- Last purchase: 2024-01-15
- Issue: Cannot reset password

User question: Why can't I reset my password?

Structured context reduces ambiguity and helps the model generate more relevant responses. It also makes your prompts easier to test and debug.

Token Budget and Response Length

Streaming doesn’t change token economics, but it changes how you think about them. Every token your model outputs costs money. In a chat application, you’re paying for every character the user sees.

Set a max_tokens parameter on every request. Sonnet 4.6 can generate up to 4,096 tokens per response by default, but most chat responses should be 500–1,500 tokens. Setting max_tokens to 1,000 not only saves money but also improves perceived latency—shorter responses stream faster.

For long-form responses (documentation, code generation, detailed explanations), consider breaking the response into chunks and making multiple requests. A user asking “write me a 10,000-word essay” will wait 30+ seconds for a single streaming response. Better to generate 2–3 shorter responses and concatenate them, or offer pagination.

Few-Shot Prompting in Streaming Contexts

Few-shot examples (showing the model examples of the desired output format) improve response quality, but they add tokens to every request. For streaming chat, use few-shot prompting sparingly and only when necessary.

If you do use examples, include them in the system message (so they’re cached across requests) or in a separate “instructions” section of the user message. Don’t repeat examples in every turn of the conversation.

Handling Conversation History

Streaming chat applications maintain conversation history. Each new user message should include the last N turns of the conversation (usually 5–10 turns, or the last 4,000 tokens of history).

Don’t include the entire conversation history. Older messages add tokens without improving response quality. Truncate history to a reasonable window, and if the user references something older, retrieve it from your database and inject it as context.

Example:

Conversation history:

User: What's the capital of France?
Assistant: The capital of France is Paris.

User: What's its population?
Assistant: Paris has a population of approximately 2.1 million people in the city proper, and over 12 million in the greater metropolitan area.

User: Tell me more about the Eiffel Tower.

Keep this structure simple and consistent. The model expects alternating user and assistant messages. If you deviate from this pattern, response quality drops.

---

Output Validation and Safety

Prompt Injection and Input Sanitisation

Streaming chat applications are vulnerable to prompt injection attacks. A user can craft a message that tricks the model into ignoring your system instructions or revealing sensitive information.

Prompt injection attacks come in two forms: direct (user input tricks the model) and indirect (injected content in retrieved documents or user data tricks the model). Both are serious in production systems.

Mitigation strategies:

1. Input validation: Reject messages longer than a reasonable limit (e.g., 10,000 characters). Long inputs are often attack attempts.
2. Content filtering: Scan user input for common injection patterns (“ignore previous instructions,” “you are now,” etc.). This isn’t foolproof but catches obvious attacks.
3. System message clarity: Make your instructions explicit and unambiguous. “You are a customer support assistant. Do not access customer data beyond what is provided in this message.” is clearer than “Be helpful.”
4. Constrained outputs: Use structured output formats (JSON, XML) where possible. It’s harder to inject malicious content into a tightly constrained format.
5. Monitoring: Log all prompts and responses. Watch for unusual patterns that might indicate attacks.

For sensitive applications (financial services, healthcare, legal), consider running a separate safety check on user input before sending it to Sonnet 4.6. This might be a simpler model or a rule-based filter.

Output Validation and Token Filtering

Streaming means you’re sending tokens to the client as they arrive. You can’t revise earlier tokens if a later token is problematic. This makes validation tricky.

Implement validation at two levels:

1. Pre-streaming validation: Before opening the stream, check that your prompt is well-formed and doesn’t violate your safety policies. This is your last chance to reject a request before tokens start flowing.
2. Token-level validation: As tokens arrive, validate them against a set of rules (no PII, no hate speech, no code injection attempts). If a token violates your rules, you have options:
   • Stop the stream: Close the connection and send an error message. The user sees an incomplete response.
   • Filter the token: Replace it with a placeholder or skip it. The user sees a garbled response.
   • Log and continue: Send the token but log it for review. This is risky but sometimes necessary.

For most applications, stopping the stream is the safest option. It’s better to show the user an error than to send harmful content.

Handling Refusals and Error States

Sonnet 4.6 sometimes refuses to answer questions (e.g., requests to help with illegal activities, generate hate speech, or reveal confidential information). When the model refuses, it sends a complete response explaining why.

In a streaming context, you need to detect refusals quickly. Some patterns to watch for:

• “I can’t help with that.”
• “I’m not able to assist with.”
• “That request violates.”
• “I don’t have the ability to.”

You can detect these patterns by checking the first few tokens of the response. If you detect a refusal, you might:

1. Show the refusal to the user: Be transparent about why the model won’t help.
2. Rephrase and retry: If the refusal seems like a false positive, rephrase the user’s question and try again.
3. Escalate to a human: For sensitive requests, route the conversation to a human agent.

Choose the approach that fits your application’s needs. For most chat applications, showing the refusal is fine.

OWASP Top 10 for LLM Applications

OWASP maintains a list of the top 10 security risks in LLM applications. If you’re building a production chat system, read this list and assess your application against each risk. The most relevant risks for streaming chat are:

• Prompt injection
• Insecure output handling
• Training data poisoning
• Unsafe plugin design
• Excessive agency

Each of these can cause real harm in production. Spend time thinking about how your application might be exploited, and implement mitigations before launch.

---

Cost Optimisation at Scale

Token Counting and Accurate Cost Forecasting

Streaming doesn’t reduce token consumption—it just distributes it over time. You still pay for every token, whether you stream it or not. But accurate token counting is harder in streaming contexts because you don’t know the final output length until the stream completes.

Anthropicprovides a stop_reason field in the final streaming event that tells you whether the response was complete (end_turn), hit the token limit (max_tokens), or stopped for another reason. Use this to understand your token consumption patterns.

Implement token counting on your backend:

import anthropic

client = anthropic.Anthropic(api_key="your-api-key")

# Count tokens before streaming
message_tokens = client.messages.count_tokens(
    model="claude-sonnet-4-6",
    messages=[
        {"role": "user", "content": "Your prompt here"}
    ]
)

print(f"Input tokens: {message_tokens.input_tokens}")

After streaming completes, log the final token count from the API response. Over time, you’ll build a profile of your typical request (input tokens, output tokens, latency). Use this to forecast costs and set budgets.

Caching and Prompt Reuse

If your chat application has a shared context that’s used across many conversations (e.g., a knowledge base, a set of system instructions, a document library), consider caching it. Anthropic supports prompt caching, which reduces token costs for repeated content.

How it works: You send a prompt with a cache_control parameter on certain blocks. Anthropic hashes the content and stores it server-side. On subsequent requests with the same content, Anthropic reuses the cached tokens at a 90% discount (input tokens cost $0.30 per 1M instead of $3).

Example:

message = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": "You are a helpful assistant."
        },
        {
            "type": "text",
            "text": "Here is a knowledge base: [large document]",
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "Query the knowledge base."}
    ]
)

Prompt caching is most effective when you have:

1. Large static context: Documents, code samples, or knowledge bases that don’t change between requests.
2. High request volume: Caching only pays off if you’re making many requests with the same cached content.
3. Repeated conversations: If users often start new conversations with the same initial context, caching saves tokens.

For a typical chat application with 1,000+ daily conversations and shared knowledge bases, prompt caching can reduce costs by 20–40%.

Batch Processing for Non-Real-Time Workloads

Not all chat requests need to be real-time. If you have background tasks (generating summaries, classifying messages, batch processing user feedback), use Anthropic’s batch API instead of the streaming API.

Batch processing is 50% cheaper than streaming but introduces latency (requests complete within 24 hours). For non-real-time workloads, it’s a no-brainer cost reduction.

Example use cases:

• Generating daily summaries of conversations
• Classifying user messages by intent or sentiment
• Extracting structured data from unstructured user input
• Generating email responses or drafts

If your application has both real-time and batch components, separate them. Route real-time requests to the streaming API and batch requests to the batch API. You’ll cut costs without sacrificing user experience.

Rate Limiting and Quota Management

At scale, you need rate limiting to prevent cost overruns and ensure fair resource allocation. Implement rate limits at multiple levels:

1. Per-user limits: Each user can make N requests per hour. Prevents abuse and controls costs.
2. Per-IP limits: Each IP address can make N requests per hour. Prevents distributed attacks.
3. Global limits: Your entire application can make N requests per hour. Prevents runaway costs.

When a user hits their rate limit, return a 429 (Too Many Requests) response and tell them when they can retry. Don’t silently queue requests—that leads to surprise bills.

Example:

from datetime import datetime, timedelta
import redis

redis_client = redis.Redis()

def check_rate_limit(user_id, limit=100, window=3600):
    key = f"rate_limit:{user_id}"
    current = redis_client.get(key)
    
    if current is None:
        redis_client.setex(key, window, 1)
        return True
    
    if int(current) >= limit:
        return False
    
    redis_client.incr(key)
    return True

Rate limits are especially important in the early days of a new chat application. You don’t know yet what your typical user looks like or how they’ll use the system. Conservative rate limits (10–50 requests per user per day) are safer than aggressive ones.

---

Common Failure Modes

Network Timeouts and Retries

Streaming requests are long-lived. A typical stream lasts 5–30 seconds. During that time, network conditions can change, connections can drop, and timeouts can occur.

Common timeout failure modes:

1. Connection timeout: The initial connection to Anthropic’s API times out. Usually a network issue on your side.
2. Read timeout: No token arrives for N seconds. The stream has stalled.
3. Client disconnect: The user closes their browser or loses connectivity. The stream is no longer needed.

Implement exponential backoff retries for timeouts:

import time
import random

def stream_with_retries(prompt, max_retries=3):
    for attempt in range(max_retries):
        try:
            return stream_response(prompt)
        except TimeoutError:
            if attempt == max_retries - 1:
                raise
            wait_time = (2 ** attempt) + random.uniform(0, 1)
            time.sleep(wait_time)

But be careful: retrying a streaming request that partially completed will double-bill you for the input tokens. Track which requests have already been partially sent, and only retry requests that haven’t started streaming yet.

Client Disconnection and Backpressure

When a user closes their browser tab, the HTTP connection closes. Your backend should detect this and stop streaming immediately. Continuing to stream after the client has disconnected wastes tokens and money.

Implement client disconnect detection:

from flask import request, Response

@app.route("/chat/stream", methods=["POST"])
def stream_chat():
    def generate():
        try:
            for token in stream_response(request.json["prompt"]):
                # Check if client is still connected
                if request.environ.get("wsgi.input").closed:
                    break
                yield token
        except GeneratorExit:
            # Client disconnected
            pass
    
    return Response(generate(), mimetype="text/event-stream")

Also implement backpressure handling. If the client’s network is slow, tokens will queue up in your send buffer. If the buffer gets too large, your server will run out of memory. Most HTTP frameworks handle this automatically, but it’s worth checking your framework’s documentation.

Model Hallucinations and Factual Errors

Sonnet 4.6 is a large language model. It sometimes generates plausible-sounding but incorrect information. In a streaming context, you can’t revise the response after it’s been sent, so hallucinations are more problematic.

Mitigation strategies:

1. Grounding in retrieved context: If your chat application has access to a knowledge base or document library, inject relevant documents into the prompt. The model is more likely to ground its response in the provided context.
2. Explicit uncertainty: Instruct the model to say “I don’t know” when uncertain. Add this to your system message: “If you’re not sure about something, say so explicitly. Do not guess or make up information.”
3. Post-generation fact-checking: After streaming completes, run the response through a fact-checking pipeline. This is expensive but important for high-stakes applications (medical, legal, financial).
4. User feedback loops: Allow users to flag incorrect responses. Use this feedback to improve your prompts and training data.

For most chat applications, grounding in retrieved context is the most effective strategy. If you don’t have relevant context to provide, consider whether the question is even appropriate for your chat assistant.

Token Limit Exceeded

If your response reaches the max_tokens limit, the stream stops abruptly. The user sees an incomplete response. This is especially common if you set max_tokens too low or the user asks for a very long response.

Handle token limit exceeded errors gracefully:

def stream_response(prompt):
    with client.messages.stream(
        model="claude-sonnet-4-6",
        max_tokens=1000,
        messages=[{"role": "user", "content": prompt}]
    ) as stream:
        for text in stream.text_stream:
            yield text
        
        # Check if we hit the token limit
        if stream.stop_reason == "max_tokens":
            yield "\n\n[Response truncated due to length. Please ask a more specific question.]\n"

Alternatively, automatically increase max_tokens if you detect that responses are frequently truncated. Monitor your logs and adjust based on actual usage patterns.

Latency Spikes and Queue Buildup

During peak traffic, Anthropic’s API might be slower than usual. Requests queue up, and time-to-first-token increases. Users see delays, and some might give up and refresh the page.

Handle latency gracefully:

1. Show a loading indicator: Let the user know the request is being processed.
2. Set realistic timeouts: If a request hasn’t started streaming after 10 seconds, show a timeout error and offer to retry.
3. Queue requests: If you’re hitting rate limits, queue requests and process them in order. Don’t drop requests silently.
4. Monitor latency: Track time-to-first-token and average response time. If latency spikes, investigate why (is the API slow? Are you overloaded? Is there a network issue?).

For most applications, latency spikes are temporary. Implement graceful degradation (show a message, offer retry) rather than crashing.

---

Real-World Implementation Patterns

Backend Stack: Python with FastAPI

Here’s a minimal production-ready streaming chat endpoint in Python:

from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse
import anthropic
import json

app = FastAPI()
client = anthropic.Anthropic(api_key="your-api-key")

@app.post("/chat/stream")
async def chat_stream(request: Request):
    body = await request.json()
    prompt = body["prompt"]
    conversation_history = body.get("history", [])
    
    # Build messages
    messages = conversation_history + [{"role": "user", "content": prompt}]
    
    # Stream response
    async def generate():
        with client.messages.stream(
            model="claude-sonnet-4-6",
            max_tokens=1000,
            system="You are a helpful assistant.",
            messages=messages
        ) as stream:
            for text in stream.text_stream:
                yield f"data: {json.dumps({'token': text})}\n\n"
            
            # Send final metadata
            yield f"data: {json.dumps({'stop_reason': stream.stop_reason})}\n\n"
    
    return StreamingResponse(generate(), media_type="text/event-stream")

This endpoint:

1. Accepts a prompt and conversation history
2. Opens a streaming connection to Anthropic
3. Yields tokens as they arrive
4. Sends final metadata when the stream completes

The client receives SSE events and renders them in real-time.

Frontend Stack: React with Fetch API

On the client side, use the Fetch API with streaming support:

async function streamChat(prompt, history) {
    const response = await fetch('/chat/stream', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ prompt, history })
    });
    
    const reader = response.body.getReader();
    const decoder = new TextDecoder();
    let buffer = '';
    
    while (true) {
        const { done, value } = await reader.read();
        if (done) break;
        
        buffer += decoder.decode(value, { stream: true });
        const lines = buffer.split('\n');
        buffer = lines.pop(); // Keep incomplete line in buffer
        
        for (const line of lines) {
            if (line.startsWith('data: ')) {
                const data = JSON.parse(line.slice(6));
                if (data.token) {
                    // Render token
                    document.getElementById('response').textContent += data.token;
                }
            }
        }
    }
}

This code:

1. Opens a streaming connection
2. Reads chunks as they arrive
3. Parses SSE events
4. Renders tokens in real-time

Database Schema for Chat History

Store conversations in a simple schema:

CREATE TABLE conversations (
    id UUID PRIMARY KEY,
    user_id UUID NOT NULL,
    created_at TIMESTAMP DEFAULT NOW(),
    updated_at TIMESTAMP DEFAULT NOW(),
    title TEXT
);

CREATE TABLE messages (
    id UUID PRIMARY KEY,
    conversation_id UUID NOT NULL REFERENCES conversations(id),
    role TEXT NOT NULL, -- 'user' or 'assistant'
    content TEXT NOT NULL,
    tokens_input INTEGER,
    tokens_output INTEGER,
    created_at TIMESTAMP DEFAULT NOW(),
    FOREIGN KEY (conversation_id) REFERENCES conversations(id)
);

CREATE INDEX idx_conversations_user ON conversations(user_id);
CREATE INDEX idx_messages_conversation ON messages(conversation_id);

When loading conversation history, fetch the last N messages (not the entire conversation). Paginate older messages on demand.

---

Testing and Observability

Unit Testing Streaming Responses

Test your streaming logic with mock responses:

import unittest
from unittest.mock import Mock, patch

class TestStreamingChat(unittest.TestCase):
    @patch('anthropic.Anthropic.messages.stream')
    def test_stream_response(self, mock_stream):
        # Mock the stream
        mock_stream.return_value.__enter__.return_value.text_stream = [
            "Hello", " ", "world", "!"
        ]
        mock_stream.return_value.__enter__.return_value.stop_reason = "end_turn"
        
        # Test your streaming logic
        result = list(stream_response("test prompt"))
        self.assertEqual(result, ["Hello", " ", "world", "!"])

Integration Testing with Real API

Test against the real Anthropic API in a staging environment:

def test_real_streaming():
    client = anthropic.Anthropic(api_key="test-key")
    
    with client.messages.stream(
        model="claude-sonnet-4-6",
        max_tokens=100,
        messages=[{"role": "user", "content": "Say hello"}]
    ) as stream:
        tokens = list(stream.text_stream)
        assert len(tokens) > 0
        assert stream.stop_reason == "end_turn"

Run integration tests daily to catch API changes or regressions.

Observability and Logging

Log every streaming request with key metrics:

import logging
import time

logger = logging.getLogger(__name__)

def stream_response(prompt):
    start_time = time.time()
    token_count = 0
    
    with client.messages.stream(
        model="claude-sonnet-4-6",
        max_tokens=1000,
        messages=[{"role": "user", "content": prompt}]
    ) as stream:
        for text in stream.text_stream:
            token_count += 1
            yield text
        
        elapsed = time.time() - start_time
        logger.info(
            "stream_completed",
            extra={
                "tokens": token_count,
                "elapsed_seconds": elapsed,
                "tokens_per_second": token_count / elapsed,
                "stop_reason": stream.stop_reason
            }
        )

Track these metrics:

• Time-to-first-token: How long before the first token arrives. Target: <500ms.
• Tokens-per-second: How fast tokens stream. Target: >20 tokens/sec.
• Total latency: Time from request to last token. Target: <10 seconds for typical responses.
• Token count: Input and output tokens. Used for cost tracking.
• Error rate: Percentage of requests that fail. Target: <1%.

Monitoring and Alerting

Set up alerts for:

1. High latency: If time-to-first-token exceeds 1 second, investigate.
2. High error rate: If >5% of requests fail, check API status.
3. Unusual token consumption: If average tokens per request spikes, check for abuse or prompt changes.
4. Cost overruns: If daily costs exceed your budget, pause new requests and investigate.

---

Moving to Production

Pre-Launch Checklist

Before launching your streaming chat application, verify:

• Rate limiting is configured – Users can’t exhaust your budget.
• Error handling is robust – Timeouts, disconnects, and API errors are handled gracefully.
• Logging and monitoring are in place – You can see what’s happening in production.
• Security measures are implemented – Input validation, prompt injection defences, output filtering.
• Cost tracking is accurate – You know how much each request costs.
• Load testing is done – You’ve tested with realistic traffic.
• Fallback mechanisms exist – If Anthropic’s API is down, users see a helpful message.
• Documentation is clear – Your team understands how the system works.

Load Testing

Simulate realistic traffic before launch:

import concurrent.futures
import time

def load_test(num_users=100, requests_per_user=10):
    def make_request():
        start = time.time()
        stream_response("What is the capital of France?")
        return time.time() - start
    
    with concurrent.futures.ThreadPoolExecutor(max_workers=num_users) as executor:
        futures = [
            executor.submit(make_request)
            for _ in range(num_users * requests_per_user)
        ]
        
        times = [f.result() for f in concurrent.futures.as_completed(futures)]
        
    print(f"Avg latency: {sum(times) / len(times):.2f}s")
    print(f"P95 latency: {sorted(times)[int(len(times) * 0.95)]:.2f}s")
    print(f"Max latency: {max(times):.2f}s")

Run load tests weekly. As your user base grows, latency will increase. Use load tests to identify bottlenecks early.

Gradual Rollout

Don’t launch to all users at once. Use a phased approach:

1. Internal testing (1 week): Your team uses the system. Find obvious bugs.
2. Beta users (2 weeks): 10–50 friendly users test the system. Gather feedback.
3. Gradual rollout (2–4 weeks): Roll out to 10% of users, then 25%, then 50%, then 100%. Monitor metrics at each step.
4. Full launch: All users have access.

At each phase, monitor error rates, latency, and cost. If something looks wrong, pause the rollout and investigate.

Incident Response

When things break in production, you need a plan:

1. Detect the issue: Monitoring alerts you to high error rates or latency.
2. Page on-call: Alert your engineering team.
3. Assess severity: Is this affecting all users or just a subset? Can users still use the application?
4. Mitigate immediately: Disable the feature if necessary. Route traffic to a fallback. Scale up resources.
5. Root cause analysis: Understand what went wrong. Was it an API issue? A code bug? A configuration error?
6. Fix and deploy: Roll out a fix during low-traffic hours.
7. Post-mortem: Document what happened and how to prevent it in the future.

Have a runbook for common incidents (API timeout, token limit exceeded, rate limit hit). Your team should be able to respond in minutes, not hours.

Optimising for Production

Once you’re live, continuously optimise:

1. Reduce latency: Profile your request path. Where are the bottlenecks? Network? Model? Your backend?
2. Reduce cost: Are there prompts you can optimise? Can you use caching more aggressively? Can you use the batch API for some workloads?
3. Improve reliability: What requests fail most often? Can you add retries or fallbacks?
4. Improve quality: Are users satisfied with responses? Can you improve prompts or add more context?

Use data from production to drive these optimisations. Don’t guess—measure.

---

Conclusion

Streaming chat with Sonnet 4.6 is achievable at scale, but it requires careful attention to architecture, prompt design, safety, and operations. The patterns in this guide are battle-tested and will serve you well in production.

Key takeaways:

1. Speed matters: Users notice every 100ms of latency. Optimise your request path ruthlessly.
2. Cost adds up: Monitor token consumption. One careless prompt can cost thousands per day at scale.
3. Safety is non-negotiable: Implement input validation, output filtering, and monitoring from day one.
4. Observability is essential: You can’t optimise what you don’t measure. Log everything.
5. Humans are in the loop: Streaming chat is a tool for humans, not a replacement. Design systems that keep humans in control.

If you’re building a production streaming chat application and need fractional engineering leadership, architectural guidance, or hands-on implementation support, PADISO’s AI & Agents Automation service can help. We’ve shipped streaming chat systems at scale and understand the patterns, pitfalls, and optimisations that work in the real world.

For teams looking to modernise their platform infrastructure alongside AI integration, PADISO’s Platform Design & Engineering service covers the full stack: API design, database schema, caching layers, and deployment infrastructure. Whether you’re in Sydney, Melbourne, New York, or any of our other offices, we bring hands-on engineering expertise and a track record of shipping production systems.

If you’re a founder or CTO without a dedicated engineering team, PADISO’s CTO as a Service provides the technical leadership and hiring expertise you need to scale. We’ve helped dozens of scale-ups build engineering teams that can ship at speed without sacrificing quality or safety.

The patterns and practices in this guide are just the starting point. Production systems are complex, and every application has unique constraints. The key is to start simple, measure everything, and iterate based on real-world data. You’ll get there.