Async Agents at Scale: Queue, Worker, Result Patterns

Introduction

When AI agents move from demos to production, the single-threaded, synchronous call-and-response pattern breaks under real workloads. An agentic system fielding 10,000 or more concurrent jobs—each potentially involving multiple model calls, tool invocations, and side effects—demands a fundamentally asynchronous architecture. Without a deliberate queue, worker, and result strategy, you court lost messages, duplicate processing, unbounded latency, and an opaque operational picture that erodes trust and ROI.

At PADISO, we’ve built and hardened async agent systems for mid‑market companies and private‑equity portfolio firms across the US, Canada, and Australia. Founder‑led by Keyvan Kasaei, our team brings fractional‑CTO depth to venture architecture and transformation—treating agent reliability not as an afterthought but as a core engineering discipline. This guide distills the patterns we’ve production‑tested: durable queuing, right‑sized worker pools, and result persistence that gives operators and business owners a clear line‑of‑sight into every job.

We’ll cover the full lifecycle from accepting a job to delivering a completed result, the infrastructure choices that matter at scale, and the observability hooks that prevent async chaos. If you’re running or planning to run agentic AI workloads on AWS, Azure, or Google Cloud, the patterns here will help you ship faster and sleep better.

Table of Contents

• Why Async Matters for Agent Systems
• Core Architecture: Queue, Workers, Results
  • The Queue: More Than Just a Buffer
  • Worker Pools: Scaling and Isolation
  • Result Persistence: What Happened After the Job
• Design Patterns for 10K+ Concurrent Jobs
  • Job Idempotency and Deduplication
  • Dead Letter Queues and Retries
  • Checkpointing and Progress Tracking
  • Rate Limiting and Backpressure
• Infrastructure Choices for Async Agents
  • Cloud‑Native Queues (AWS SQS, Azure Service Bus, Google Cloud Pub/Sub)
  • Redis and PostgreSQL as Job Stores
  • When to Use Workflow Engines (Temporal, Dagster)
• Observability and Debugging
  • Distributed Tracing and Metrics
  • Logging Correlation and Error Recovery
• Integrating with Agentic AI Frameworks
  • Async Patterns with Claude Opus 4.8, Sonnet 4.6
  • Orchestrating Multi‑Agent Workloads
• How PADISO Delivers Reliability at Scale
• Summary and Next Steps

Why Async Matters for Agent Systems

An AI agent often performs a chain of actions: it reasons, calls external APIs, waits for human approval, or spawns sub‑agents. These steps are inherently long‑running—measured in seconds to minutes, not milliseconds. If your front‑end or orchestrator blocks on the full execution, you quickly exhaust thread pools and degrade user experience. Worse, transient failures in downstream services or model timeouts can cascade into lost work.

Moving to an async model with a durable queue decouples the acceptance of a task from its execution. This immediately improves throughput and fault tolerance. It also enables horizontal scaling: you can add worker replicas independently of the submission rate, and you can pause or throttle workers during maintenance without accepting failures.

For the CEO or board member of a $50M–$250M company, async isn’t a technical nicety—it’s a prerequisite for AI ROI. When every job ties back to revenue operations, customer experience, or supply‑chain decisions, lost or stuck jobs are lost money. As we tell private‑equity operating partners considering a roll‑up tech consolidation, the async foundation is what turns a promising AI pilot into a value‑creation lever across portfolio companies.

Core Architecture: Queue, Workers, Results

The canonical async agent pattern has three components: a durable queue that holds pending and in‑flight jobs, a pool of worker processes that consume jobs and execute agent logic, and a result store where completed outputs and metadata are persisted. The following sequence diagram illustrates the flow for a single job submission.

sequenceDiagram
    participant Client
    participant API Gateway
    participant Queue
    participant Worker
    participant Result Store
    Client->>API Gateway: POST /jobs (idempotency key)
    API Gateway->>Queue: Enqueue job with metadata
    API Gateway-->>Client: 202 Accepted + job ID
    Worker->>Queue: Poll for job
    Queue-->>Worker: Dequeue job
    Worker->>Worker: Execute agent workflow (model calls, tools)
    Worker->>Result Store: Write result + status
    Client->>API Gateway: GET /jobs/{id}
    API Gateway->>Result Store: Fetch result
    API Gateway-->>Client: 200 OK with result

Let’s examine each piece.

The Queue: More Than Just a Buffer

A queue is responsible for receiving, storing, and delivering job messages to workers. At 10K+ concurrent jobs, the queue must guarantee at‑least‑once delivery, provide message visibility timeouts, and support dead‑letter routing for poison messages. It must also offer low latency—a few tens of milliseconds from enqueue to first poll—so workers aren’t idle.

For cloud‑native deployments, managed services like AWS SQS, Azure Service Bus, and Google Cloud Pub/Sub provide these guarantees with minimal operational overhead. We’ll compare them in the infrastructure section. The key design decisions are:

• FIFO vs. standard: For most agent jobs, strict ordering isn’t required, but out‑of‑order execution can cause confusion when jobs depend on prior state. When ordering matters, use a FIFO queue or partition on a business key.
• Visibility timeout: Set it just longer than the expected maximum processing time. If a worker crashes, the message reappears after the timeout and is picked up by another worker. Too short, and you get duplicate processing; too long, and stalled jobs block throughput.
• Message size limits: If your job payload exceeds the queue’s limit (e.g., 256 KB for SQS), store the payload in an object store like S3 and put a reference in the queue message.

Many agent frameworks treat the queue as a simple transport. Production readiness means treating it as a critical piece of infrastructure with monitoring on queue depth, age of oldest message, and dead‑letter accumulation. The guide on deploying AI agents with async workers from Railway illustrates this pattern with Redis and PostgreSQL—a pragmatic choice we’ve also used for smaller deployments.

Worker Pools: Scaling and Isolation

Workers are the processes that pull jobs, execute the agent, and push results. The design of the worker pool determines how well the system scales and how gracefully it handles failures.

Pool sizing is not guesswork. For I/O‑bound workloads dominated by model API calls, a worker can comfortably handle many concurrent jobs using asynchronous I/O (e.g., Python’s asyncio or Node.js event loop). For CPU‑bound tasks—such as local inference or heavy data processing—you’ll want a pool of OS threads or separate processes. The Scaling Swarms documentation provides guidance on worker count tuning: start with 2–4 workers per vCPU for I/O‑bound workloads, then scale based on queue depth and latency percentiles.

Isolation matters because agents can be unpredictable. A single job that consumes excessive memory or triggers an infinite loop shouldn’t bring down the whole pool. Container‑based workers (Kubernetes pods, ECS tasks) with resource limits give you per‑job sandboxing. If a worker exceeds its memory or time budget, it’s killed and the job is retried or sent to the dead‑letter queue.

Gracious shutdown is mandatory. When a deployment rolls out a new image, existing workers should finish their current jobs before terminating. Most queue libraries support a heartbeat or deleteMessage on successful completion, so a clean shutdown avoids message loss.

Result Persistence: What Happened After the Job

Once a worker completes a job, the result must be stored durably and made queryable. A common mistake is to push the result back into a response queue and rely on the client to retrieve it via a callback. At scale, that creates tight coupling and complicates error handling.

A better pattern is a dedicated result store—a database table or key‑value store indexed by job ID. The worker writes the final state and any output payload. The client polls or subscribes to updates. This decoupling makes the system more resilient and enables long‑term auditability.

For agent systems, the result isn’t just a binary blob; it often includes an execution trace: which models were called, which tools were invoked, what intermediate decisions were made. We recommend storing a structured JSON document in a relational database (PostgreSQL) or a document store (MongoDB). This trace is invaluable for AI strategy and readiness assessments and for debugging compliance issues—for example, when demonstrating chain‑of‑custody for a financial decision to a regulator.

The asynchronous API pattern—returning HTTP 202 Accepted and a status endpoint—is described thoroughly in the guide on designing async, long‑running APIs for AI agents. We standardize on this pattern because it aligns with REST conventions and makes it straightforward to build dashboards that show job completion rates and error clusters.

Design Patterns for 10K+ Concurrent Jobs

With the basic building blocks in place, scaling to 10,000+ concurrent jobs requires deliberate patterns for reliability, cost control, and observability.

Job Idempotency and Deduplication

At scale, network glitches and client retries mean you will receive duplicate job requests. An idempotent job design ensures that processing the same logical request twice yields the same outcome and doesn’t cause double billing or data corruption.

Assign an idempotency key on the client side—often a UUID or a hash of the business parameters. The API gateway checks a fast cache (Redis) to see if the key has already been processed. If yes, return the existing job ID and status. If no, enqueue the job and record the key. This pattern is further detailed in the async agent workflows guide, which emphasizes idempotency keys and checkpointing.

Critically, idempotency must span the entire processing window. For long‑running agent jobs that may exceed cache TTLs, use a persistent store and set the cache with a generous expiry—or store the mapping in the same database as the result.

Dead Letter Queues and Retries

Not all failures are transient. A malformed input, an expired API key, or a deterministic bug in a tool will cause a job to fail repeatedly. Without a dead‑letter queue (DLQ), these poison messages clog the main queue and trigger endless retries, wasting worker capacity and model tokens—directly eating into AI ROI.

Configure a separate DLQ with a limited retention period. After a configurable number of retries (we typically use 3–5), send the message there. Set up monitoring and alerting on DLQ depth, and provide operators a manual replay mechanism once the root cause is fixed.

The AgentStack async patterns checklist recommends exponential backoff for retries and emphasizes that each retry should increase the delay to avoid thundering herds. In our own deployments, we combine immediate retry for transient network errors with delayed retries for rate‑limited API responses.

Checkpointing and Progress Tracking

Multi‑step agent workflows can take minutes. If a worker crashes halfway through, you don’t want to restart from scratch—that wastes model calls and time. Checkpointing stores intermediate state so that a worker can resume from the last completed step.

Implement checkpointing at logical boundaries in the agent’s plan. For a planner‑worker separation pattern, the planner first creates a list of tasks, and the worker executes them one by one, updating a completed_steps counter in the result store. The planner‑worker separation pattern is a clean way to isolate exploration from execution, and it makes resumption simple: a worker picks up a job and skips tasks already marked complete.

Checkpointing also improves the user experience. A progress endpoint that returns “steps completed / total” transforms an opaque, long‑running process into a dashboard‑friendly metric—exactly the kind of transparency that heads of engineering at mid‑market firms demand.

Rate Limiting and Backpressure

External APIs—model providers, SaaS tools—enforce rate limits. Without client‑side throttling, your workers will hammer these APIs, get 429 responses, and waste capacity in retries. A proper async system implements backpressure: when the downstream system signals congestion, workers slow down.

Use a token‑bucket or leaky‑bucket algorithm per API endpoint. Store rate‑limit state in Redis so that all workers share a global view. When approaching limits, workers should voluntarily delay or pause polling. This is where async truly shines: the queue acts as a shock absorber, allowing spikes in job submissions without overwhelming dependencies.

Operations guides like Concurrency, Queues & Scaling describe fair dispatcher queues and per‑tenant caps for sub‑agents—a pattern we implement for multi‑tenant SaaS agent platforms built on AWS and Azure.

Infrastructure Choices for Async Agents

The queue and worker stack you choose has a direct impact on maintenance cost, latency, and scaling limits. We’ll compare three broad options.

Cloud‑Native Queues (AWS SQS, Azure Service Bus, Google Cloud Pub/Sub)

For teams already committed to a hyperscaler, cloud‑native queues offer the simplest path to production. They’re fully managed, support at‑least‑once delivery, and integrate natively with the provider’s identity and monitoring.

• AWS SQS: Great for high‑throughput, standard‑queue workloads. FIFO queues provide ordering but have a 300‑message‑per‑second ​limit per partition. Combine with Lambda or ECS workers for serverless or container‑based processing. PADISO frequently uses SQS in platform engineering for San Francisco Bay Area startups.
• Azure Service Bus: Strong choice in Microsoft‑centric shops with native dead‑lettering and session support for ordered processing. Its premium tier offers higher throughput and predictable latency.
• Google Cloud Pub/Sub: Ideal for globally distributed workloads. Push subscriptions can deliver directly to a worker endpoint, but pull subscriptions with flow control give more backpressure control.

A common pitfall: managed queues have default limits that can be exceeded by 10K+ concurrent jobs. You must request quota increases and configure queue partitioning appropriately.

Redis and PostgreSQL as Job Stores

For teams that want a multi‑cloud or simpler stack, Redis‑backed queues (Resque, Bull, Celery with Redis broker) and PostgreSQL‑backed queues (pg‑boss, Oban) are mature options. They add operational responsibility—you must run and scale the backing store—but avoid vendor lock‑in.

Redis delivers sub‑millisecond latency and excellent throughput for small messages. Its List and Stream data structures make it a natural fit for job queuing. We often recommend Redis for early‑stage startups before moving to cloud‑native queues as traffic grows, a path we’ve followed with teams in Austin and Edmonton.

PostgreSQL‑based queues are appealing because you already have a database. They provide transactional enqueue/dequeue, so you can atomically insert a job and its initial state. This simplifies idempotency and auditability. The downside: poll‑based workers add latency and database load at high concurrency. For 10K+ concurrent jobs, dedicated queue infrastructure is usually necessary.

When to Use Workflow Engines (Temporal, Dagster)

Long‑running, complex workflows with many steps, human‑in‑the‑loop approvals, and compensation logic benefit from a full workflow engine. Temporal and Dagster provide durable execution, eliminating the need to build custom checkpointing and retry logic.

Temporal is particularly strong for agent orchestration. It persists every event and lets you replay a workflow from any point, which is invaluable for debugging agent decisions. However, it introduces an additional infrastructure component and a learning curve. We typically recommend it for enterprise deployments where compliance and auditability are requirements—exactly the profile of a firm undergoing SOC 2 or ISO 27001 audit readiness.

For purely batch‑oriented agent pipelines, Dagster’s declarative approach to data pipelines can be extended to run agent steps, but it lacks the general‑purpose workflow capabilities of Temporal. The async agent workflows blog explores trade‑offs between Temporal and a queue‑plus‑database approach; we advise teams to use the heaviest tool only when the workflow complexity justifies it.

Observability and Debugging

Async systems are harder to debug than synchronous ones because the call stack is fragmented across processes, machines, and time. Robust observability is non‑negotiable.

Distributed Tracing and Metrics

Instrument each job with a trace context that propagates from the client request through the queue to the worker and into model calls. OpenTelemetry makes this straightforward. Key metrics to track:

• Queue depth and age of oldest message: Indicates backlog. P95 processing time should stay within your SLO.
• Worker utilization and error rates: Alert on spikes.
• DLQ depth: Should be near zero. A growing DLQ is an immediate signal that something is broken.
• End‑to‑end latency percentiles: From job submission to result available.

We build dashboards that surface these metrics for every portfolio company we support through our CTO as a Service engagement. A well‑instrumented system gives the board confidence that the AI investment is delivering measurable throughput.

Logging Correlation and Error Recovery

Stamp every log line with the job ID. Use structured logging (JSON) so that you can aggregate logs from multiple workers and search for a single job’s lifecycle. When a job fails, the error log should include the failed step, the input payload, and the error from the downstream service—enough to reproduce the issue without re‑running the entire workflow.

Pair this with an error‑recovery playbook. For common transient failures, the system should auto‑retry. For deterministic failures, alert an operator and surface the job in a “stalled jobs” UI. The agent orchestration patterns article from Michael Pena emphasizes idempotency, timeouts, and the need for a supervisor pattern to monitor worker health—something we encode directly into our platform engineering for Melbourne‑based insurance and health teams.

Integrating with Agentic AI Frameworks

Async Patterns with Claude Opus 4.8, Sonnet 4.6

Modern LLMs like Claude Opus 4.8 and Sonnet 4.6 are too capable to be limited by synchronous request‑response cycles. When an agent uses these models for chain‑of‑thought reasoning, the model’s native thinking budget can keep a worker busy for tens of seconds. Meanwhile, competing offerings like GPT‑5.6 (Sol and Terra) or Kimi K3 also require the same async treatment under load.

We standardize on a pattern where each model call within a worker is a discrete, idempotent step. The worker stores the model’s response in the checkpoint, so if the worker crashes, it doesn’t re‑invoke the model. This small optimization saves thousands of dollars in token costs at scale—a detail that directly improves AI ROI for our clients.

Orchestrating Multi‑Agent Workloads

When a high‑level planner dispatches sub‑tedits to specialist agents, the async architecture becomes hierarchical. The planner enqueues sub‑tasks, each of which flows through its own queue‑worker‑result lifecycle. A top‑level result is only complete when all sub‑results are assembled.

We recommend the Agent Orchestration Patterns for designing the supervisor/worker separation. The supervisor tracks dependencies and handles timeouts. This pattern maps directly to the planner‑worker separation we reference earlier.

How PADISO Delivers Reliability at Scale

At PADISO, we don’t just advise on architecture—we build and operate it. Our Venture Architecture & Transformation practice implements these patterns for companies that can’t afford downtime or lost jobs. For a recent mid‑market logistics client, we migrated their prototype agent from a monolithic FastAPI service to an async architecture on AWS SQS and ECS, enabling them to handle 8,000 concurrent jobs during peak season with a p99 latency under 12 seconds.

For a private‑equity roll‑up consolidating three SaaS platforms, we standardized the async agent stack on Azure Service Bus and Temporal, cutting infrastructure costs by 30% while improving auditability for SOC 2 compliance. These are the kinds of results we deliver through our fractional CTO engagements in New York, Los Angeles, Seattle, and beyond.

If you’re scaling an agent system or planning a multi‑workload AI transformation, we bring the operator’s perspective that demo‑stage startups or pure consulting shops often lack. Our work spans platform development in Adelaide for sovereign environments, AI advisory in Sydney, and financial‑services AI that is compliant by design. See our case studies for concrete outcomes.

Summary and Next Steps

Async architectures for agent systems aren’t optional once you move beyond a prototype. A production‑grade queue, worker, and result design gives you the throughput, durability, and observability needed to run 10,000+ concurrent jobs safely. The key patterns to remember:

• Decouple with a durable queue and use 202 Accepted plus a status endpoint.
• Right‑size worker pools based on workload profile and impose hard resource limits.
• Persist results with full execution traces and make them queryable.
• Design for idempotency, dead‑letter routing, and checkpointing from day one.
• Choose infrastructure that matches your scale and operational maturity—cloud‑native queues for hyperscaler shops, Redis/Postgres for simplicity, Temporal for complex workflows.
• Instrument everything with distributed tracing and structured logs.

Implementing these patterns is what separates a promising AI pilot from a reliable, high‑ROI production system. If your team is evaluating AI strategy and readiness, scaling platform engineering, or seeking a fractional CTO to own the technical direction, PADISO is built to partner. Reach out to learn how we bring production‑tested async architectures to mid‑market brands and private‑equity portfolios—on AWS, Azure, Google Cloud, and beyond.