Apache Superset Performance: Long-Running Reports

Table of Contents

1. Why Long-Running Reports Matter
2. Understanding Superset’s Query Execution Model
3. Async Queries: The Foundation of Production Stability
4. Database-Level Tuning for Performance
5. Caching Strategies That Actually Work
6. Resource Allocation and Infrastructure
7. Monitoring and Observability
8. Real-World Configuration Patterns
9. Common Pitfalls and How to Avoid Them
10. Next Steps: Moving to Production

---

Why Long-Running Reports Matter

Long-running reports are the backbone of serious analytics operations. When you’re dealing with 100GB+ datasets, complex joins across multiple fact tables, or rolling aggregations spanning years of data, your dashboard and reporting layer can’t rely on sub-second query responses. The organisations that get this right—and ship reliable analytics—treat long-running reports as a first-class operational concern, not an edge case.

At PADISO, we’ve worked with teams across financial services in Sydney, government operations in Canberra, and enterprise retail operations who all faced the same problem: Superset timeouts, hung queries, user frustration, and eventually, a retreat to manual SQL notebooks. The difference between those who solved it and those who didn’t came down to three things: understanding Superset’s async execution model, configuring it correctly, and building operational habits around monitoring and resource limits.

This guide captures the real patterns we apply on customer engagements—concrete configurations, specific numbers, and the operational discipline that separates production Superset deployments from demo environments.

---

Understanding Superset’s Query Execution Model

Before you tune anything, you need to understand how Superset executes queries. Out of the box, Superset runs queries synchronously: a user clicks a dashboard, Superset sends a query to the database, and the browser waits for the result. If that query takes 5 minutes, the browser connection hangs for 5 minutes. If it takes longer than your web server’s timeout (often 30–60 seconds), the query dies and the user sees an error.

For small datasets and simple queries, this works fine. For long-running reports—anything that touches millions of rows, requires complex window functions, or aggregates across years of historical data—synchronous execution is a recipe for failure.

Superset’s architecture includes three key layers:

The Web Server (Flask): Handles HTTP requests, validates SQL, and orchestrates query execution. It’s stateless and horizontally scalable, but it can’t hold connections indefinitely.

The Query Executor: The component that actually sends queries to your database and waits for results. In synchronous mode, this blocks the web server. In async mode, it offloads to a task queue.

The Result Backend: Where query results are cached and stored temporarily. This is typically Redis or a database table, and it’s critical for async queries because the web server needs somewhere to stash results while the query runs.

The key insight: async queries decouple query execution from the HTTP request cycle. When you enable async mode, Superset queues the query, returns immediately to the user, and lets a worker process handle the actual database call. The user polls for results, and when they’re ready, they’re served from the result backend.

This is not a Superset-specific pattern. It’s how every production analytics platform handles long-running workloads. The challenge is configuring it correctly and understanding the trade-offs.

---

Async Queries: The Foundation of Production Stability

Async queries are non-negotiable for production Superset deployments handling long-running reports. Here’s how to set them up and what to expect.

Enabling Async Query Execution

Async queries require a task queue (Celery, typically) and a result backend (Redis or a database). If you’re running Superset in production, you should already have both. If not, start here: Apache Superset Documentation: Async Queries covers the full setup.

Once you have the infrastructure, enable async in your superset_config.py:

# Enable async query execution
ASYNC_QUERY_JOB_TIMEOUT = 3600  # 1 hour timeout for long-running queries
SUPERSET_WEBSERVER_TIMEOUT = 60  # HTTP timeout; doesn't block async queries

# Configure the task queue
CELERY_BROKER_URL = "redis://redis:6379/0"
CELERY_RESULT_BACKEND = "redis://redis:6379/1"

# Enable async for all queries (or scope it to specific databases)
ASYNC_QUERY_MANAGER_CELERY_ASYNC_TIME_LIMIT_SEC = 3600

# Polling configuration for the frontend
SUPERSET_ASYNC_QUERY_POLL_INTERVAL = 2000  # milliseconds between polls

The key numbers here are:

• ASYNC_QUERY_JOB_TIMEOUT: Set this to the longest query you expect to run. We typically use 3600 seconds (1 hour) for large aggregations, 1800 seconds (30 minutes) for standard reports. Don’t set it arbitrarily high; it’s a safety valve against runaway queries.
• SUPERSET_WEBSERVER_TIMEOUT: This is the HTTP timeout for synchronous requests. Keep it short (30–60 seconds) so that slow queries fail fast and trigger async fallback.
• SUPERSET_ASYNC_QUERY_POLL_INTERVAL: The frontend polls for results every 2 seconds by default. For reports that take minutes, this is fine. For very long reports (30+ minutes), you might increase it to 5000ms to reduce polling overhead.

Monitoring Async Queue Depth

Once async is enabled, you need visibility into the queue. If your Celery queue is backing up, users will see “pending” states for minutes, and you need to know about it.

Add this to your monitoring:

# In a custom metrics endpoint
from celery import current_app

def get_queue_stats():
    inspector = current_app.control.inspect()
    active_tasks = inspector.active()
    reserved_tasks = inspector.reserved()
    
    total_active = sum(len(v) for v in active_tasks.values())
    total_reserved = sum(len(v) for v in reserved_tasks.values())
    
    return {
        "active_queries": total_active,
        "pending_queries": total_reserved,
    }

If your pending queue regularly exceeds 10–20 queries, you need more workers. We typically run 1 Celery worker per 2–4 concurrent users, depending on average query runtime. For a team of 20 analysts, that’s 5–10 workers.

Result Backend Sizing

Async queries store results in Redis (or your result backend). Each result is typically 10MB–100MB for large reports. If you have 50 concurrent users and each query produces a 50MB result, you need at least 2.5GB of result backend capacity.

Monitor Redis memory usage:

redis-cli INFO memory

If you’re consistently above 80% capacity, either increase Redis memory or implement result eviction. We typically set Redis to evict oldest results first:

maxmemory-policy allkeys-lru

---

Database-Level Tuning for Performance

Async queries only work if your database can actually execute them efficiently. This is where most teams stumble. They enable async, then blame Superset when queries still take 10 minutes.

The reality: Superset is a query executor, not a query optimizer. If your underlying database is slow, Superset will be slow. This is especially true for long-running reports that touch large tables.

Indexing Strategy for Analytical Queries

Analytical queries are different from transactional queries. Your OLTP indexes (unique keys, foreign keys on high-cardinality columns) don’t help analytics.

For long-running reports, create composite indexes on the columns you filter and group by:

-- If your report filters by date and groups by region
CREATE INDEX idx_events_date_region 
ON events(event_date, region_id);

-- If you're doing time-series aggregations
CREATE INDEX idx_metrics_timestamp_metric_id 
ON metrics(timestamp, metric_id);

The order matters. Put the most selective filter first, then the grouping column. If you’re unsure, check your query plan:

EXPLAIN ANALYZE SELECT * FROM events 
WHERE event_date > '2024-01-01' 
GROUP BY region_id;

Look for “Seq Scan” on large tables. If you see it, you need an index.

Partitioning for Scale

Once your fact tables exceed 1–2 billion rows, full-table scans become prohibitively slow. Partitioning is the answer.

For time-series data (events, metrics, logs), partition by date:

CREATE TABLE events (
    id BIGINT,
    event_date DATE,
    region_id INT,
    value DECIMAL(10, 2)
) PARTITION BY RANGE (YEAR(event_date)) (
    PARTITION p_2022 VALUES LESS THAN (2023),
    PARTITION p_2023 VALUES LESS THAN (2024),
    PARTITION p_2024 VALUES LESS THAN (2025),
    PARTITION p_future VALUES LESS THAN MAXVALUE
);

When a report filters on event_date > '2024-01-01', the database only scans the relevant partitions. For a 5-year dataset, this can reduce scan time from 30 minutes to 2 minutes.

We’ve seen teams at financial services platforms in New York cut long-running report times by 40–60% just by partitioning their fact tables.

Materialized Views and Aggregation Tables

Some reports are inherently slow because they require complex aggregations. Don’t try to optimize them away—pre-compute them.

If you have a report that aggregates 2 years of events by day and region, create a materialized view:

CREATE MATERIALIZED VIEW events_daily_region AS
SELECT 
    event_date,
    region_id,
    COUNT(*) as event_count,
    SUM(value) as total_value,
    AVG(value) as avg_value
FROM events
WHERE event_date >= DATE_SUB(CURDATE(), INTERVAL 2 YEAR)
GROUP BY event_date, region_id;

CREATE INDEX idx_events_daily_region_date ON events_daily_region(event_date);

Refresh this view daily (or hourly, depending on freshness requirements):

REFRESH MATERIALIZED VIEW events_daily_region;

Now, reports that query this view run in seconds instead of minutes. The trade-off: the data is stale until the next refresh. For most analytics use cases, daily or hourly freshness is acceptable.

Connection Pooling and Query Limits

Each Superset query opens a database connection. If you have 50 Celery workers and each runs a query, that’s 50 simultaneous connections. Most databases can handle this, but it adds overhead.

Configure connection pooling in your database URI:

# In superset_config.py
SQLALCHEMY_ENGINE_OPTIONS = {
    "pool_size": 10,
    "max_overflow": 20,
    "pool_recycle": 3600,
    "echo_pool": False,
}

These numbers mean:

• pool_size: Keep 10 idle connections open. Reuse them for new queries.
• max_overflow: Allow up to 20 additional connections beyond pool_size if needed.
• pool_recycle: Close connections older than 1 hour. This prevents stale connections.

For long-running reports, also set query timeouts at the database level:

# In superset_config.py, per database
SQLALCHEMY_CONNECT_ARGS = {
    "connect_timeout": 30,
    "command_timeout": 3600,  # 1 hour
}

If a query exceeds the timeout, the database kills it and Superset reports an error. This prevents hung connections from accumulating.

---

Caching Strategies That Actually Work

Caching is the most underutilised lever in Superset. Most teams enable it, set a 1-hour TTL, and call it done. That’s leaving performance on the table.

Query-Level Caching

Superset caches query results automatically. When a user runs the same query twice, the second execution hits the cache and returns instantly.

Configure cache timeouts in superset_config.py:

# Cache all queries for 1 hour by default
CACHE_DEFAULT_TIMEOUT = 3600

# But override for specific databases or tables
SUPERSET_CACHE_TIMEOUT = {
    "default": 3600,
    "realtime_db": 300,  # 5 minutes for real-time data
    "warehouse_db": 86400,  # 24 hours for data warehouse
}

For long-running reports, longer cache timeouts are usually fine. If a report takes 10 minutes to run, caching it for 24 hours is reasonable—most analytical questions don’t change minute-to-minute.

Dashboard-Level Caching

Dashboards with multiple charts can be slow because Superset executes all charts serially (or in parallel, depending on configuration). If each chart takes 2 minutes and you have 5 charts, the dashboard takes 10 minutes to load.

Cache entire dashboards:

# In superset_config.py
DASHBOARD_CACHE_TIMEOUT = 3600  # Cache dashboard state for 1 hour

This caches the entire rendered dashboard, not just individual queries. Users see the cached version instantly. The trade-off: if someone updates a chart, other users see the stale version for up to 1 hour.

For dashboards that are viewed frequently and updated rarely, this is a massive win. We’ve seen dashboards that used to take 5 minutes to load now load in under 1 second.

Cache Invalidation Strategy

Caching without invalidation is dangerous. If a report is cached and the underlying data changes, users see stale results.

Implement cache invalidation rules:

# Invalidate cache when data is updated
from superset.extensions import cache_manager

def on_data_update(table_name):
    # Clear cache for all queries on this table
    cache_manager.delete_memoized(f"table_{table_name}")

For long-running reports, we recommend:

• Cache for 24 hours if data is updated once daily (e.g., nightly ETL).
• Cache for 1 hour if data is updated multiple times daily.
• Cache for 5 minutes if data is updated continuously (real-time dashboards).
• No cache for regulatory or compliance reports that must always reflect current state.

Redis Configuration for Caching

Superset’s cache backend (usually Redis) needs tuning for analytical workloads:

# In superset_config.py
CACHE_CONFIG = {
    "CACHE_TYPE": "redis",
    "CACHE_REDIS_URL": "redis://redis:6379/2",
    "CACHE_REDIS_DB": 2,
    "CACHE_KEY_PREFIX": "superset_",
}

# Redis memory management
REDIS_CONFIG = {
    "maxmemory": "4gb",
    "maxmemory-policy": "allkeys-lru",  # Evict least-recently-used keys
}

Monitor cache hit rates:

redis-cli INFO stats | grep hits

A healthy cache has a hit rate above 70%. If yours is below 50%, either increase Redis memory or adjust cache timeouts.

---

Resource Allocation and Infrastructure

Long-running reports require infrastructure that can handle them. This section covers the operational decisions that separate production deployments from demo environments.

Celery Worker Sizing

Celery workers execute queries. Each worker can run one query at a time. If you have 5 workers and 10 concurrent users, 5 queries queue up and wait.

Size your worker pool based on:

Workers = (Concurrent Users × Average Query Runtime) / Acceptable Queue Wait Time

Example: 20 concurrent users, 5-minute average query runtime, 30-second acceptable wait:

Workers = (20 × 5 × 60) / 30 = 200 worker-seconds needed
Workers = 200 / 60 ≈ 4 workers

In practice, we typically run 1 worker per 2–4 concurrent users. For a team of 20, that’s 5–10 workers. For enterprise deployments, we’ve run 50+ workers.

Configure worker concurrency in your Docker Compose or Kubernetes setup:

services:
  superset-worker:
    image: superset:latest
    command: celery --app=superset.tasks worker --loglevel=info --concurrency=4
    environment:
      - CELERY_BROKER_URL=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/1
    depends_on:
      - redis
      - postgres

Memory and CPU Allocation

Superset’s web server and workers need different resources.

Web Server (Flask):

• 2–4 CPU cores
• 2–4GB RAM
• Scales horizontally; run 2–3 instances behind a load balancer

Celery Workers:

• 1–2 CPU cores per worker
• 1–2GB RAM per worker (queries can be memory-intensive)
• Scales based on concurrent query load

Result Backend (Redis):

• 1 CPU core
• Memory = (Concurrent Queries × Average Result Size) × 2
• For 20 concurrent queries, 50MB results: 2GB RAM

Database Connection Pool:

• Typically 10–20 connections per Superset instance
• Each connection consumes ~1MB on the database
• For 3 Superset instances with 10 connections each, that’s 30 connections total

We worked with trading and logistics teams in Chicago who initially undersized their result backend. They had 50+ concurrent users running 5-minute queries, and Redis was constantly evicting results. Users would get “result not found” errors. Adding 4GB of Redis memory solved the problem immediately.

Database Replica for Analytics

Long-running analytical queries can slow down your production database. The solution: run Superset against a read replica.

# In superset_config.py
SQLALCHEMY_DATABASES = {
    "production": {
        "engine": "mysql",
        "sqlalchemy_uri": "mysql://user:pass@prod-db:3306/mydb",
    },
    "analytics": {
        "engine": "mysql",
        "sqlalchemy_uri": "mysql://user:pass@replica-db:3306/mydb",
    },
}

Point all Superset dashboards at the analytics database (the replica). This isolates analytical load from transactional load.

For large organisations, we recommend a dedicated data warehouse (Snowflake, BigQuery, Redshift) for analytics, entirely separate from production databases. This is standard practice at financial services platforms and retail operations we’ve worked with.

---

Monitoring and Observability

You can’t optimise what you can’t measure. Monitoring is non-negotiable for production Superset deployments.

Key Metrics to Track

Query Execution Time:

• P50, P95, P99 latency
• Percentage of queries exceeding timeout
• Slow query log (queries > 5 minutes)

Queue Depth:

• Active queries
• Pending queries
• Average wait time in queue

Resource Utilisation:

• Celery worker CPU and memory
• Redis memory usage and eviction rate
• Database connection pool usage
• Database CPU and I/O during peak query load

Cache Performance:

• Cache hit rate
• Cache eviction rate
• Average cache entry size

Instrumentation

Add custom logging to Superset to capture query metadata:

# In superset_config.py
import logging
from pythonjsonlogger import jsonlogger

logger = logging.getLogger("superset.queries")
logHandler = logging.FileHandler("superset_queries.log")
formatter = jsonlogger.JsonFormatter()
logHandler.setFormatter(formatter)
logger.addHandler(logHandler)

# Log query execution
def log_query_execution(query, duration_seconds, result_rows):
    logger.info("query_executed", extra={
        "query_hash": hash(query),
        "duration_seconds": duration_seconds,
        "result_rows": result_rows,
        "timestamp": datetime.utcnow().isoformat(),
    })

Parse these logs with a tool like ELK Stack or Datadog. You’ll quickly identify which queries are slow and which tables need optimisation.

Alerting

Set up alerts for common problems:

# Example Prometheus alert rules
groups:
  - name: superset
    rules:
      - alert: HighQueryQueueDepth
        expr: superset_pending_queries > 20
        for: 5m
        annotations:
          summary: "Superset query queue depth > 20"
      
      - alert: HighQueryLatency
        expr: histogram_quantile(0.95, superset_query_duration_seconds) > 300
        for: 10m
        annotations:
          summary: "P95 query latency > 5 minutes"
      
      - alert: RedisMemoryHigh
        expr: redis_memory_used_bytes / redis_memory_max_bytes > 0.85
        for: 5m
        annotations:
          summary: "Redis memory usage > 85%"

---

Real-World Configuration Patterns

This section captures the exact configurations we use on customer engagements. These are battle-tested patterns that work.

Pattern 1: High-Volume Analytics (1000+ daily active users)

Setup:

• 3 Superset web servers behind a load balancer
• 20 Celery workers
• PostgreSQL with read replicas for analytics
• Redis (8GB) for caching and result backend
• ClickHouse or Snowflake for data warehouse

Configuration:

# superset_config.py
ASYNC_QUERY_JOB_TIMEOUT = 3600
SUPERSET_WEBSERVER_TIMEOUT = 60
SUPERSET_ASYNC_QUERY_POLL_INTERVAL = 2000

SQLALCHEMY_ENGINE_OPTIONS = {
    "pool_size": 20,
    "max_overflow": 30,
    "pool_recycle": 3600,
}

CACHE_DEFAULT_TIMEOUT = 3600
DASHBOARD_CACHE_TIMEOUT = 3600

CELERY_BROKER_URL = "redis://redis:6379/0"
CELERY_RESULT_BACKEND = "redis://redis:6379/1"

Expected Performance:

• P50 query latency: 5–10 seconds
• P95 query latency: 30–60 seconds
• P99 query latency: 2–5 minutes
• Cache hit rate: 75%+
• Queue wait time: < 30 seconds (average)

Pattern 2: Regulatory / Compliance Reporting

Setup:

• Single Superset instance (no horizontal scaling required)
• 4–6 Celery workers
• Dedicated PostgreSQL instance (no read replicas; must reflect current state)
• Redis (2GB) for result backend only (no caching for compliance reports)
• Audit logging for all queries

Configuration:

# superset_config.py
ASYNC_QUERY_JOB_TIMEOUT = 1800  # 30 minutes max
SUPERSET_WEBSERVER_TIMEOUT = 60

SQLALCHEMY_ENGINE_OPTIONS = {
    "pool_size": 10,
    "max_overflow": 10,
    "pool_recycle": 3600,
}

# No caching for compliance queries
CACHE_DEFAULT_TIMEOUT = 0

# Audit logging
SUPERSET_LOG_QUERY_EXECUTION = True
SUPERSET_AUDIT_LOG_ENABLED = True

Expected Performance:

• Query latency: 1–10 minutes (acceptable for compliance)
• All queries logged and auditable
• No stale data

Pattern 3: Real-Time Operational Dashboards

Setup:

• 2 Superset web servers
• 8 Celery workers
• Data warehouse (Snowflake, BigQuery) with real-time data ingestion
• Redis (4GB) for aggressive caching (5-minute TTL)
• WebSocket support for live updates

Configuration:

# superset_config.py
ASYNC_QUERY_JOB_TIMEOUT = 300  # 5 minutes (real-time queries must be fast)
SUPERSET_WEBSERVER_TIMEOUT = 30
SUPERSET_ASYNC_QUERY_POLL_INTERVAL = 1000  # Poll every 1 second

CACHE_DEFAULT_TIMEOUT = 300  # 5-minute cache
DASHBOARD_CACHE_TIMEOUT = 300

# Enable WebSocket for live updates
SUPERSET_ENABLE_WEBSOCKET = True

Expected Performance:

• Dashboard load time: 1–3 seconds (cached)
• Chart update latency: 5–10 seconds (new data)
• Cache hit rate: 80%+

---

Common Pitfalls and How to Avoid Them

Pitfall 1: Timeout Cascades

Problem: Web server timeout (60s) is longer than database timeout (30s). When a query times out at the database, Superset waits another 30 seconds for a response that will never come.

Solution: Always set web server timeout shorter than database timeout:

SUPERSET_WEBSERVER_TIMEOUT = 30  # Web server
DATABASE_QUERY_TIMEOUT = 3600   # Database

When the database kills a query at 30 seconds, the web server immediately reports the error instead of waiting.

Pitfall 2: Unbounded Result Sets

Problem: A user writes a query that returns 1 million rows. Superset tries to return all of them, exhausts memory, and crashes.

Solution: Enforce result limits:

# In superset_config.py
SUPERSET_ROW_LIMIT = 100000  # Max rows per query
SUPERSET_SAMPLES_ROW_LIMIT = 10000  # Max rows for data preview

If a user’s query would exceed the limit, Superset truncates the result and displays a warning.

Pitfall 3: Missing Indexes on Filter Columns

Problem: A report filters on created_at > '2024-01-01' but there’s no index on created_at. Superset scans 1 billion rows and times out.

Solution: Index all columns used in WHERE clauses:

CREATE INDEX idx_events_created_at ON events(created_at);

Run EXPLAIN ANALYZE on every report query and check for full table scans. If you see one, add an index.

Pitfall 4: Async Queries Without Monitoring

Problem: Async is enabled, but nobody monitors the Celery queue. Queries pile up, users wait 30 minutes, and you don’t know why.

Solution: Monitor queue depth and alert on thresholds:

def monitor_queue():
    inspector = celery_app.control.inspect()
    active = sum(len(v) for v in inspector.active().values())
    reserved = sum(len(v) for v in inspector.reserved().values())
    
    if reserved > 20:
        send_alert("Superset query queue depth > 20")

Pitfall 5: Cache Invalidation Bugs

Problem: Data is updated, but the cache isn’t invalidated. Users see stale results for hours.

Solution: Implement cache invalidation in your data pipeline:

# In your ETL/data update process
from superset.extensions import cache_manager

def update_events_table():
    # Update the table
    execute_sql("INSERT INTO events ...")
    
    # Invalidate cache for all queries on this table
    cache_manager.delete_memoized("table_events")

Alternatively, use short cache timeouts (5 minutes) for frequently updated tables.

---

Next Steps: Moving to Production

You’ve read the patterns. Here’s how to actually ship this.

Phase 1: Assessment (Week 1)

1. Identify your long-running reports. Which dashboards take > 2 minutes to load?
2. Profile your database. Run EXPLAIN ANALYZE on each report query. Look for full table scans and missing indexes.
3. Audit your current Superset configuration. Is async enabled? What’s your cache strategy?
4. Measure baseline performance. P50, P95, P99 query latency. Cache hit rate. Queue depth.

Phase 2: Infrastructure (Weeks 2–3)

1. Set up Celery and Redis if not already in place.
2. Configure connection pooling and query timeouts.
3. Create a read replica or data warehouse for analytics queries.
4. Deploy monitoring and alerting.

Phase 3: Database Optimisation (Weeks 3–4)

1. Add indexes on filter and grouping columns.
2. Implement partitioning for tables > 1 billion rows.
3. Create materialized views for complex aggregations.
4. Test and measure improvements.

Phase 4: Superset Tuning (Week 4)

1. Enable async queries with appropriate timeouts.
2. Configure caching with realistic TTLs.
3. Tune Celery worker count based on concurrent load.
4. Size Redis and connection pools.

Phase 5: Validation (Week 5)

1. Load test with realistic query patterns.
2. Verify all alerts fire correctly.
3. Document runbooks for common issues.
4. Train your team on monitoring and troubleshooting.

Getting Help

If you’re running Superset at scale and hitting performance walls, this is exactly the kind of problem PADISO’s platform engineering teams solve. We’ve optimised Superset deployments for financial services, government, and enterprise retail organisations across Australia and the US.

Our approach:

1. Audit your current setup (database, Superset config, infrastructure).
2. Profile your actual query patterns and identify bottlenecks.
3. Optimise at the database and application level.
4. Validate with load testing and production monitoring.
5. Handoff with runbooks and team training.

We typically see 40–70% latency improvements and 60%+ cache hit rates after optimisation. The work is specific to your data, queries, and infrastructure—there’s no one-size-fits-all answer.

If you’re ready to ship production Superset with long-running reports, book a call with our platform engineering team. We’ll assess your setup, identify quick wins, and build a roadmap to production stability.

---

Summary

Long-running reports are hard. They require async query execution, database optimisation, careful resource allocation, and relentless monitoring. But the payoff is enormous: reliable analytics, happy users, and a platform that actually scales.

The key takeaways:

1. Async queries are mandatory. Enable Apache Superset’s async query execution and size your Celery workers appropriately.
2. Database optimisation is foundational. Indexes, partitioning, and materialized views will save you more time than any Superset configuration.
3. Caching works, but only with discipline. Set realistic TTLs, implement invalidation, and monitor hit rates.
4. Infrastructure matters. Right-size your Redis, database connections, and worker pools based on concurrent load and query runtime.
5. Monitor everything. Query latency, queue depth, cache hit rate, database load. You can’t optimise what you can’t measure.

Follow these patterns, and you’ll ship Superset deployments that handle long-running reports reliably and at scale. Teams at platform development across Australia and the United States are doing this right now—and so can you.