Table of Contents
- Why Your Agent System Needs a Durable Task Queue
- How FOR UPDATE SKIP LOCKED Works
- Production-Grade Task Queue Patterns
- Throughput, Concurrency, and Scaling
- Failure Modes and How to Defend Against Them
- How PADISO Deploys This in AI Agent Systems
- Next Steps: From Postgres Queue to Autonomous Workflows
Why Your Agent System Needs a Durable Task Queue
Modern AI agent systems—whether orchestrating Claude Opus 4.8 to reason over a knowledge base, delegating tasks to Sonnet 4.6 and Haiku 4.5 sub-agents, or coordinating multi-step automation with GPT-5.6 Sol and Terra—live and die by reliable task distribution. When you have a dozen agents that need to share work without stepping on one another, in-memory queues and ephemeral job dispatchers become a liability. The moment a container restarts, a node scales down, or a worker crashes, you lose not just the job, but often the visibility into what went wrong. This is where the humble Postgres database—already the backbone of most applications—steps up with a surprisingly powerful capability: FOR UPDATE SKIP LOCKED.
The Hidden Cost of In-Memory Job Dispatch
Startups often start with Redis or RabbitMQ for job queues, and that works until it doesn’t. Redis can lose jobs on a crash if not configured with AOF and proper persistence. RabbitMQ adds an extra moving part that needs to be monitored and hardened. For teams shipping AI features under tight deadlines, every new service is a new attack surface and a new compliance burden—especially relevant when you’re pursuing SOC 2 or ISO 27001 audit-readiness via Vanta. At PADISO, we’ve seen mid-market companies and private-equity portfolios repeatedly hit the wall with orphaned tasks, inconsistent state, and hours of manual reconciliation. The answer, more often than not, is to leverage a database you already trust and operate: Postgres.
Where Postgres Fits In
Postgres is a rock-solid, transactional database that just happens to include robust row-level locking. With FOR UPDATE SKIP LOCKED, it can behave like a durable, crash-safe task queue without any additional infrastructure. This pattern powers agentic AI workloads on platform development in New York, Chicago, and San Francisco where a PADISO-led CTO as a Service engagement streamlines everything from LLM orchestration to background data processing. When a private equity roll-up wants to consolidate tech stacks across 14 portfolio companies, we often start by building a central task database using Postgres SKIP LOCKED—it’s auditable, familiar to every DBA, and eliminates a redundant message broker. That’s real EBITDA lift from infrastructure simplification.
How FOR UPDATE SKIP LOCKED Works
At its core, FOR UPDATE SKIP LOCKED combines two concepts: row-level locks that prevent other transactions from modifying a row, and a “skip” directive that makes a transaction ignore rows already locked by another concurrent transaction. Instead of waiting or raising an error, the query simply moves on to the next available row. This is the atomic baton-pass that lets multiple workers pull tasks from the same table without a central arbiter.
The SQL That Powers Atomic Claims
Imagine a tasks table with a status column and a claimed_by field. Many workers run this single statement:
WITH cte AS (
SELECT id
FROM tasks
WHERE status = 'pending'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 1
)
UPDATE tasks
SET status = 'processing',
claimed_by = current_setting('app.worker_id'),
started_at = now()
FROM cte
WHERE tasks.id = cte.id
RETURNING tasks.*;That’s it. In one atomic step, a worker finds an unclaimed task, locks it, updates it, and receives the full row—all within a single transaction. No two workers ever claim the same row because the lock is exclusive. The official guide on using FOR UPDATE SKIP LOCKED for queue workflows describes exactly how this pattern eliminates contention and keeps the queue moving.
Why SKIP LOCKED Is the Key
Without SKIP LOCKED, a concurrent FOR UPDATE would block until the first transaction commits or rolls back, turning your parallel workers into a serial bottleneck. By skipping locked rows, each worker immediately grabs the next available task, achieving true parallelism with minimal overhead. This is the same atomic guarantee you’d get from a dedicated queue like Kafka or SQS, but without the extra infrastructure. A detailed Prisma blog post breaks down how a Postgres job queue with SKIP LOCKED can replace Redis entirely, and we’ve applied that insight on platform development in Seattle for cloud-native startups that run entirely on Aurora.
A Quick Mermaid Flow
Here’s how two workers safely claim tasks from a shared table:
sequenceDiagram
participant W1 as Worker 1
participant DB as Postgres
participant W2 as Worker 2
Note over DB: tasks table with pending rows
W1->>DB: SELECT ... FOR UPDATE SKIP LOCKED LIMIT 1
DB-->>W1: returns task #1, locks row
W2->>DB: SELECT ... FOR UPDATE SKIP LOCKED LIMIT 1
Note over DB: task #1 is locked, so skip it
DB-->>W2: returns task #2 (next available)
W1->>DB: UPDATE set status='processing'
W2->>DB: UPDATE set status='processing'
Both workers proceed independently.No central dispatcher, no race conditions, no lost work.
Production-Grade Task Queue Patterns
Getting the basic claim working is just the beginning. A production agent system needs robustness across crashes, reprocessing, and idempotency. These patterns are battle-tested from our work on platform development in Toronto where Canadian fintechs require PIPEDA-aware durability, and on platform development in Melbourne for scale-ups that refuse to accept job loss during deploys.
State Machine and Status Columns
Use an enum or a check-constrained varchar for status. A common set: pending, processing, completed, failed, dead. The FOR UPDATE SKIP LOCKED query only looks at pending, so workers never touch tasks that are already being processed or are finished. This is the simplest way to enforce exactly-once delivery at the queue layer.
A production guide for PostgreSQL job queues using SKIP LOCKED walks through a four-state machine that also handles orphan jobs and replay, and we incorporate those lessons into every AI agent pipeline we architect.
Lease/Heartbeat for Crash Resiliency
What if a worker grabs a task and then crashes? The row stays locked until the transaction rolls back (if the connection breaks), but you don’t want to wait for a TCP timeout. Instead, add a lease_expires_at column. Workers periodically update a heartbeat_at or extend the lease inside their processing transaction. A separate reaper job can reclaim tasks whose lease has lapsed, resetting them to pending. The Dropless.dev guide on Postgres as a queue implements a robust lease/heartbeat mechanism with partial indexes that keeps the reaper overhead minimal. At PADISO, we bake this into our AI & Agents Automation engagements—every agent worker registers a unique ID via current_setting('app.worker_id') and renews its lease every few seconds.
Idempotency Keys for Exactly-Once Processing
Even with atomic claiming, a worker might process a task successfully but fail to commit the transaction (e.g., network blip). To prevent duplicate execution, add an idempotency_key column. Before any side effects, the worker checks whether an outcome with that key already exists. This is critical when your agents are making API calls to Claude Opus 4.8, Sonnet 4.6, or GPT-5.6 Sol because re-running a model invocation wastes tokens and can produce inconsistent downstream state.
Orphan Job Reclamation
The reaper process mentioned earlier is an essential component for platform development in Atlanta where PCI-aware fintech workloads can’t afford to have a single job stall. The reaper runs on a schedule (say, every 30 seconds) and updates tasks whose lease_expires_at < now() back to pending. To avoid contention, it uses FOR UPDATE SKIP LOCKED just like the workers. This self-healing loop ensures that even if a Kubernetes node is terminated mid-inference, the task will be retried by a healthy agent.
Throughput, Concurrency, and Scaling
It’s common to ask: can Postgres really keep up with hundreds of worker agents? The answer is yes, and often better than you’d expect. When built correctly, a SKIP LOCKED queue can sustain thousands of task claims per second on modest hardware because the lock overhead is minimal and the index scans are efficient.
How Many Workers Can You Run?
Postgres’s connection limit is typically the bottleneck before CPU. With a connection pooler like PgBouncer, you can scale to hundreds of workers. Each worker holds a transaction only for the duration of the claim, which is milliseconds, then releases the connection. We’ve seen 50–100 concurrent workers on a single RDS db.r5.large instance easily handle mixed workloads. For larger fleets, platform development in United States often uses reader endpoints and horizontal read scaling, but the claim query itself must hit the primary. The Stack Overflow discussion on atomic UPDATE … SELECT in Postgres includes community benchmarks that show linear scaling up to many workers with SKIP LOCKED.
Indexing and Partial Indexes for Hot Pathways
A naive scan on WHERE status = 'pending' will slow down as the table grows. The solution is a partial index:
CREATE INDEX idx_tasks_pending_created
ON tasks (created_at)
WHERE status = 'pending';This keeps the index small and fast, exactly covering the rows the claim query cares about. The reaper can use a similar partial index on WHERE status = 'processing' AND lease_expires_at < now(). These tuning steps are standard in our Platform Design & Engineering practice and consistently deliver sub-millisecond fetch times even with millions of completed tasks.
LISTEN/NOTIFY for Near-Real-Time Dispatch
Polling the database every second works, but you can do better with Postgres’s built-in pub/sub. After inserting a new task, the producer issues NOTIFY tasks_channel. Workers connect once, listen for notifications, and query immediately upon receiving one. This reduces latency from 1 second to single-digit milliseconds without adding a Redis instance. The Habr article on implementing a task queue with SKIP LOCKED and lease/heartbeat shows how Russians platforms use FOR NO KEY UPDATE SKIP LOCKED combined with LISTEN/NOTIFY for high-frequency trading-style workloads. We adapt the same pattern for platform development in Gold Coast where tourism applications need real-time booking orchestration across multiple agentic services.
Failure Modes and How to Defend Against Them
Even a well-designed queue will encounter edge cases. The mark of a production system is how it handles them. Here are the most common failure modes and the defensive layers we embed in every PADISO engagement.
Worker Panics and Lost Locks
A worker that crashes without cleaning up its lock will cause the task to remain in processing indefinitely unless the lease reaper intervenes. That’s why the lease pattern is mandatory. Additionally, you should set a statement_timeout at the session level to prevent a runaway query from holding a lock forever. SET statement_timeout TO '5min'; before processing begins ensures that the lock won’t outlive your entire worker pool.
Connection Storms Under Load
When a flood of tasks arrives, every worker attempts to claim simultaneously, potentially overwhelming the connection pool. To prevent this, use a backoff strategy: on receiving an empty result from the claim query (no pending tasks), wait with exponential backoff up to a cap. PgBouncer’s transaction pooling mode also reduces the number of idle connections drastically. The article on why Node.js SaaS needs a persistent task queue illustrates how a single misconfigured worker can flood the database, and we use those lessons when setting up agent workers for platform development in Sydney.
Transaction Wraparound and Long-Running Claims
A task that takes 10 hours to complete (e.g., a long agent loop with multiple LLM calls) can be problematic because it keeps a transaction open. Consider breaking the work into smaller tasks or using a status update pattern: the worker claims the task, updates status to processing, commits immediately, then re-checks the task’s liveness periodically with separate transactions. This avoids transaction ID wraparound risks and lets the reaper see an up-to-date lease. For heavy agent workflows, we often use a combination of a quick-claim queue and a longer-running executor state machine, a pattern refined during platform development in Perth where mining telemetry pipelines run for hours.
Checklists for Production Readiness
Before putting a FOR UPDATE SKIP LOCKED queue into production, ensure:
- Partial indexes on
statuscolumns - Lease/heartbeat with reaper job
- Idempotency keys enforced at the application layer
- Connection pooling with PgBouncer
- Monitoring: queue depth, claim rate, reaper activity, average processing time
- Graceful shutdown for workers (commit or rollback cleanly)
- Tested recovery from database failover: does the new primary correctly handle in-flight locks?
This checklist is part of our Security Audit (SOC 2 / ISO 27001) readiness engagements, because a task queue that loses jobs is a compliance finding waiting to happen.
How PADISO Deploys This in AI Agent Systems
When we step in as fractional CTOs for a mid-market company or a private equity roll-up, the task queue pattern often becomes the linchpin of the entire AI platform. For example, a recent project with a portfolio of e-commerce brands needed a way to coordinate 30 AI agents—some powered by Claude Sonnet 4.6 for product tagging, others by GPT-5.6 Sol for customer service auto-responses, and a handful using Kimi K3 for multilingual translation. Each agent needed to pull work from a single queue without collisions.
We built a Postgres-backed agent_tasks table with FOR UPDATE SKIP LOCKED claims, a lease/heartbeat with a 60-second expiry, and a reaper that runs every 15 seconds. The result: zero lost tasks, predictable throughput even when we scaled from 3 to 30 workers, and a clean audit trail for the Vanta compliance dashboard. That same architecture now supports platform development in Brisbane where logistics companies use agentic automation to dispatch fleet instructions.
We also lean on this pattern in our Venture Studio & Co-Build engagements, where time-to-ship is measured in weeks. Instead of building a complex microservice mesh with Kafka and Kafka Connect, we stand up a simple task table with LISTEN/NOTIFY and idempotent processors. That decision alone cuts infrastructure costs by 40% and speeds up launch by 6 weeks—exactly the kind of outcome a board or PE operating partner wants to hear.
If you’re evaluating how to scale AI agents across your portfolio, take a look at our case studies for real results on efficiency consolidation and AI ROI.
Next Steps: From Postgres Queue to Autonomous Workflows
FOR UPDATE SKIP LOCKED is a gateway drug to building robust, durable, explainable agent systems. Once you have a reliable task queue, you can layer on:
- Observability: log every state transition to a separate
task_eventstable for dashboards (we use Superset + ClickHouse for this, as detailed in our platform development in Austin work). - Prioritization: add a
prioritycolumn andORDER BY priority DESC, created_atin the claim query. - Dynamic routing: use a
task_typefield to route work to different worker pools (each with its owncurrent_setting('app.worker_type')filter). - Scheduled tasks: create a
run_atcolumn and base the pending condition onrun_at <= now().
At PADISO, we help teams go from a single Postgres table to a full AI Strategy & Readiness execution in under a quarter. Whether you’re a founder who needs a fractional CTO to lead the build, or a private equity firm driving a roll-up consolidation and AI value creation, this pattern is a safe, battle-tested starting point.
Reach out if you’re ready to move from ad-hoc job dispatch to production-grade agent orchestration. We’ll help you design the schema, tune the throughput, and get your AI agents shipping measurable results—without ever losing a task.