Table of Contents
- Why Superset and Iceberg?
- The D23.io Advantage
- Core Connection Patterns
- Query Performance Tuning
- Operational Quirks and How We Solved Them
- Security and Compliance
- Case Studies: D23.io in Action
- Summary and Next Steps
Why Superset and Iceberg?
Apache Superset and Apache Iceberg have become the open-source power couple for modern analytics, and for good reason. Superset delivers pixel-perfect dashboards with no per-seat licensing trap, while Iceberg brings ACID transactions, time travel, and seamless schema evolution to your data lake. Together, they enable what every mid-market CEO, PE operating partner, and startup founder wants: fast, self-serve analytics on a single source of truth, without the $500k Tableau renewal. At PADISO, we’ve shipped this stack in dozens of customer deployments through our D23.io product line, and we’ve learned what actually works in production.
The typical use case is predictable: a company has a petabyte-scale data lake on S3, teams want to visualize it without copying data into a proprietary warehouse, and the existing BI tool is either too expensive or too slow. Iceberg solves the table format headache; Superset solves the visualization and self-service exploration. But connecting them cleanly, at scale, is where the rubber meets the road.
The D23.io Advantage
D23.io is our opinionated reference implementation for embedded analytics and internal BI stacks. It bundles Superset, ClickHouse, and Iceberg into a hardened deployment package that we tailor for each client’s hyperscaler footprint—be it AWS, Azure, or Google Cloud. What sets D23.io apart is the operational know-how baked in from hundreds of production hours. We’ve already solved the connection patterns, the caching headaches, and the security hurdles, so you don’t have to burn a sprint figuring out why Trino’s memory pool is spilling to disk.
When you engage PADISO for a fractional CTO or CTO as a Service engagement, D23.io becomes a strategic asset: a pre-built analytics backbone that turns months of integration work into weeks of configuration. Our platform engineering teams in Chicago, Dallas, San Francisco, Sydney, and Brisbane have deployed it across finance, logistics, manufacturing, and retail, each time adapting the stack to local data gravity and compliance requirements.
Core Connection Patterns
The first architectural decision you’ll face is how Superset talks to Iceberg. There are three proven patterns, each with its own trade-offs. We’ll walk through them with the nuance you won’t find in a vendor whitepaper.
flowchart TD
subgraph Clients ["User Access"]
A[Superset Dashboard]
end
subgraph Query Layer ["Query & Cache"]
B[Trino Coordinator]
C[(Redis Cache)]
D[Superset Metadata DB]
end
subgraph Data Layer ["Data Storage"]
E[Iceberg Tables]
F[(S3 / ADLS / GCS)]
G[Hive Metastore]
end
A -->|SQL via SQLAlchemy| B
A -->|Dashboard configs| D
B -->|Read metadata| G
B -->|Read Parquet files| F
B -->|Cache query results| C
G -->|Table locations| F
E -->|Writes via Spark/Flink| FTrino as the Gateway
This is the most battle-tested approach and the default in D23.io. Trino acts as a federated query engine, speaking Iceberg natively and pushing down predicates to S3. Superset connects to Trino via SQLAlchemy, and from there you get a single pane of glass across multiple catalogs. We typically configure a separate Trino catalog per Iceberg namespace—iceberg.dwh for core dimensional data, iceberg.events for high-velocity event streams—and set up resource groups to prevent one misbehaving dashboard from starving others.
The connector config is straightforward:
connector.name=iceberg
iceberg.catalog.type=hive
hive.metastore.uri=thrift://metastore:9083
iceberg.file-format=PARQUETBut the real work is in tuning the Trino <-> Iceberg handshake. Iceberg’s manifest files are the secret sauce for partition pruning, and Trino can skip entire files if the query predicates align with the table’s partition spec. This is why we insist on time-based partitioning for fact tables: a dashboard filtering on order_date = '2024-03-15' will scan a few megabytes instead of a few terabytes.
One operational quirk: if you’re using Iceberg’s metadata_location to point at the latest snapshot, Trino may cache an old manifest list. We’ve seen dashboards show stale data until the coordinator is restarted, so we now set iceberg.catalog.cached-catalog-expiration to a TTL that matches your ETL pipeline’s SLA. This is the kind of production scar tissue you get when you run this stack at scale, and it’s why our case studies emphasize observability.
Direct S3 Access: Parquet to the People
An emerging alternative is to skip the query engine and have Superset directly query Parquet files via DuckDB or DataFusion. DuckDB can scan Iceberg metadata and execute SQL on S3 objects, and Superset can use DuckDB as a backend. This pattern slashes infrastructure cost—no Trino cluster to manage—and can be surprisingly performant for small to medium datasets (< 500 GB). In fact, our platform engineering team in Canada recently used this approach for a logistics client that needed embeddable analytics but couldn’t justify a dedicated query tier.
The trade-off is concurrency. DuckDB is single-node, so if you have 50 dashboard users hammering the same table, you’ll hit I/O limits quickly. We mitigate this with a read-only replica pattern: an Iceberg table is synced hourly to a set of optimized Parquet files, and DuckDB instances are deployed behind a load balancer. It’s not fully ACID, but for many operational analytics workloads, eventual consistency is good enough.
Hybrid Approaches
The most flexible deployments use Trino for complex ad-hoc queries and DuckDB for frequently accessed dashboards. We’ve built a materialization layer in D23.io that populates a ClickHouse table from Iceberg snapshots; Superset then queries ClickHouse directly, achieving sub-second response times for the top 10% of queries. This is the pattern we recommend for private equity roll-ups where the goal is portfolio value creation through consolidated analytics. The cost savings from decommissioning legacy BI tools often fund the entire platform modernization.
Query Performance Tuning
Performance isn’t a feature; it’s the product. A dashboard that takes 30 seconds to load is one nobody uses. Here’s how we tune the Superset-to-Iceberg path to keep latency below 3 seconds for 95% of queries.
Partition Pruning and Metadata Optimization
Iceberg’s hidden partitioning means you don’t need to know the partition column to get pruning; the engine does it automatically. But you still need to choose a good partition transformation. For timestamp columns, days(ts) works well for daily dashboards; for multi-tenant tables, identity(tenant_id) or bucket(tenant_id, 16) prevents excessive partitioning. We regularly VACUUM orphan files and compact small files using Apache Spark jobs—iceberg’s rewriteDataFiles action can reduce query scan time by 40% in our benchmarks.
A less visible bottleneck is the Hive Metastore (HMS). If your Trino cluster is hammering HMS for partition metadata on every query, latency spikes. We co-locate a Thrift-based HMS with the Trino coordinator and enable iceberg.catalog.cached-catalog with a 10-minute cache. For truly large deployments, AWS Glue Data Catalog with automatic statistics generation can keep query plans efficient without manual intervention.
Superset-Specific Tuning
Superset’s SQL Lab can be your best friend or your worst enemy. We enforce row limits and query timeouts at the database level—never rely on Superset’s own settings to prevent a runaway query from melting your Trino cluster. Our standard policy: MAX_ROW_LIMIT = 50000, SQL_MAX_ROW = 50000, and a query timeout of 120 seconds. For dashboards, we pre-warm the most common filter combinations using a cron job that runs curl against the dashboard’s permalink. This keeps cache hit rates high. Speaking of caching, let’s dive into the layers.
Caching Layers
Superset offers three caching levels: query (the result set from SQL Lab), chart data (the JSON payload for a viz), and dashboard metadata. We use Redis for all three, with a TTL of 300 seconds for chart data and 86400 seconds for metadata. But the real win is in Trino’s own result cache. By enabling query.cache.enabled=true and pointing it to a Redis instance, Trino can short-circuit entire query executions if the underlying table files haven’t changed. Since Iceberg snapshot IDs are monotonic, we set a cache key that includes the snapshot ID; this invalidates naturally when new data lands.
For the ultimate in performance, we implement a materialized view layer using dbt and ClickHouse. dbt models transform Iceberg data into denormalized star schemas, which are then ingested into ClickHouse tables via a lightweight Apache Flink job. Superset connects to ClickHouse and serves dashboards in milliseconds. This pattern is at the heart of many D23.io deployments and is detailed in our platform development guides for US clients.
Operational Quirks and How We Solved Them
No architecture survives contact with production. Here are the quirks we’ve encountered and the fixes we’ve baked into D23.io.
Schema Evolution Surprises
Iceberg’s schema evolution is a superpower: add a column, drop a column, rename a column—all without rewriting the table. But Superset’s metadata layer doesn’t always keep up. If you add a new column to an Iceberg table and immediately try to chart it in Superset, you’ll often get a “column not found” error until you manually edit the table’s schema in Superset’s data source editor. We’ve automated this with a post-deploy hook that calls the Superset API to refresh the datasource after every schema change. Another gotcha: renaming a column in Iceberg breaks existing Superset charts silently. We version our table schemas and maintain a compatibility view using Trino’s create view until all dashboards are migrated.
Time-Travel Queries and Security
Iceberg’s time travel (FOR VERSION AS OF or FOR TIMESTAMP AS OF) is an auditor’s dream, but it can expose data that was supposed to be deleted for GDPR reasons. We audit-all time-travel access by routing it through a dedicated Trino catalog that enforces row-level policies. Superset users never see the time-travel SQL; we build a separate “Audit” dashboard that uses a read-only service account with SELECT permissions on historical snapshots. This keeps the compliance team happy without slowing down business users.
Monitoring and Alerting
You can’t tune what you don’t measure. We ship D23.io with a Prometheus/Grafana stack that monitors Trino query latency, resource group queue depths, Iceberg metadata operations per second, and Superset dashboard load times. Alerts fire when the 95th percentile latency exceeds 10 seconds for more than 5 minutes. A common failure mode is a sudden spike in iceberg.manifest_list_cache.misses—this usually means a compaction job failed and the table has accumulated too many small files. Our blog details the observability playbook.
Security and Compliance
Mid-market firms and PE portfolios often face SOC 2 or ISO 27001 requirements, and analytics pipelines are in scope. Here’s how we harden the stack.
Fine-Grained Access Control
Trino’s native authorization integrates with Open Policy Agent (OPA) or Apache Ranger to enforce column- and row-level security. We map Superset user roles to Trino user groups, so a sales manager in the Dallas office sees only their region’s data. Iceberg tables are stored in S3 with bucket policies that allow only the Trino service role to read—no direct S3 access for end users. For multi-tenant SaaS platforms, we use Iceberg’s branch feature to create per-customer tables from a shared master, a technique we’ve refined in our platform engineering practice across Australia.
Audit Readiness with Vanta
PADISO uses Vanta to automate evidence collection for SOC 2 and ISO 27001 audits. For the analytics stack, Vanta monitors S3 bucket policies, Trino access logs, and Superset authentication events. If a user enables time travel without authorization, the infraction is flagged automatically. This audit readiness gives PE firms confidence that a roll-up’s data platform won’t derail a deal’s compliance due diligence. Our about page goes deeper into our security-first ethos.
Case Studies: D23.io in Action
Mid-Market Consolidation for a PE Roll-Up
A PE firm’s portfolio company in logistics had grown through five acquisitions, each running a different BI tool on a different data warehouse. PADISO stepped in with a fractional CTO and a rapid AI transformation mandate. Using D23.io, we consolidated all data into a single Iceberg lake on S3, deployed Trino as the unified query engine, and rolled out Superset dashboards for operational KPIs across all subsidiaries. The result: a 60% reduction in analytics TCO and a 2-second average query latency, directly contributing to a 12% EBITDA lift within six months. Read more in our case studies.
Startup Embedding Analytics
A seed-stage startup building a customer-facing analytics product needed to embed Superset without the operational burden. D23.io’s Helm chart and Terraform modules allowed them to deploy a multi-tenant instance on Google Cloud Platform in under a day. We used DuckDB for direct S3 queries to keep costs near zero, and the caching layer ensured snappy performance even with a free-tier Superset instance. The startup’s CEO cited our CTO as a Service as the reason they could ship analytics features three months faster than planned.
Summary and Next Steps
The Superset-on-Iceberg reference architecture we’ve shipped through D23.io is production-grade, cost-efficient, and ready for the demands of mid-market scale-ups and PE consolidations. The key takeaways:
- Choose a connection pattern that fits your data size and concurrency: Trino for robustness, DuckDB for simplicity, or a hybrid with ClickHouse for speed.
- Invest in partition pruning, metadata caching, and file compaction to keep queries fast.
- Bake in observability from day one; latency spikes are rarely Superset’s fault, but an Iceberg compaction gap.
- Security isn’t optional—leverage Trino’s fine-grained access and Vanta for audit readiness.
Ready to see how PADISO can accelerate your analytics modernization? Book a call with our platform engineering team. Whether you’re rolling up a portfolio in Chicago, launching a data product in Sydney, or aiming for SOC 2 compliance for your SaaS platform, D23.io gives you the architecture that wins deals.
Explore more on our services page or connect with us on LinkedIn. For real-time updates and deep dives, follow our blog.