Superset on Kubernetes: Upgrade and Recovery Drills That Matter

A Helm release can be rolled back while the Superset metadata database remains changed. That mismatch is the central reason to rehearse upgrades as recovery drills, not treat them as routine chart updates. A useful drill proves that your team can identify the release and database versions, restore a compatible metadata backup, bring the intended application version back online, and verify that users see correct data and access.

This walkthrough focuses on that sequence. It assumes Superset is already running on Kubernetes and that your deployment uses Helm. The commands use placeholders for release-specific names and versions; replace them with values from your own cluster and change process. Run recovery commands only in an isolated rehearsal environment until your team has validated the exact procedure for production.

The Superset upgrade guidance calls for backing up metadata and checking migration compatibility and recovery before an upgrade. A Helm rollback changes the selected Helm release revision, but it does not reverse a database migration or other external side effects. Those are separate recovery tasks. (Superset upgrade guidance; Helm rollback documentation)

1. Prerequisites: make the drill repeatable

Before scheduling an upgrade exercise, ensure the team can identify every version and dependency that matters. At minimum, record the Helm release name and namespace, chart version, Superset image tag or digest, current configuration revision, metadata database endpoint, and the backup identifier that the drill will use. Include the relevant worker image and configuration if workers are deployed separately. A tag such as stable is not a sufficiently precise recovery reference if its contents can change.

You need access to the Kubernetes context used for the rehearsal, Helm, kubectl, a metadata backup and a means to restore it to a separate database, and a Superset image that your organization has approved for the test. Confirm that the operator can read release history and workload status. Keep passwords and connection strings out of command-line arguments, terminal transcripts, and article examples; use your organization’s established secret-injection process.

Provision an isolated namespace and, preferably, an isolated metadata database. Do not point the drill at production simply because the chart values are convenient. A recovery rehearsal that restores over live metadata can become an outage rather than a test. The rehearsal should use a copy of configuration with production credentials and sensitive endpoints removed or replaced, while preserving the structural settings needed to exercise the same migration path.

Choose a maintenance window for the exercise and name one operator, one observer, and one person authorized to stop it. Decide in advance what constitutes a stop condition: for example, an unexpected migration failure, a restore to the wrong database, a chart rendering change that alters an unreviewed resource, or a failed access-control check. This is a proposed operating practice, not a Superset or Helm feature.

A drill must also have a known-good reference. Keep a record of the last application and chart combination that passed your release checks, along with its compatible configuration and a metadata backup from before the upgrade. A chart revision alone is not a complete reference if the image, secret values, external database, or configuration changed independently.

2. Set up a safe release boundary

Use the same chart repository and release naming conventions as the live deployment, but resolve the specific chart version and image version that the drill will test. Do not assume that the latest chart is the right target. Review the chart’s values and rendered resources for your chosen versions, and save those artifacts with the change record so a later operator can see what was actually proposed.

Create a small environment file containing non-secret identifiers. For example:

export NAMESPACE='superset-rehearsal'
export RELEASE='superset-rehearsal'
export TARGET_CHART_VERSION='REPLACE_WITH_REVIEWED_CHART_VERSION'
export CURRENT_CHART_VERSION='REPLACE_WITH_CURRENT_CHART_VERSION'
export TARGET_IMAGE='REPLACE_WITH_APPROVED_IMAGE_REFERENCE'

These values are illustrative. The correct chart reference, image settings, and value keys depend on the release and chart version in use. Check the chart’s values for those versions rather than copying an old configuration into a new release without review. Keep secrets in a protected values file or secret manager; do not place them in this environment file.

Create the rehearsal namespace and confirm that the active context is the intended one:

kubectl config current-context
kubectl create namespace "$NAMESPACE"
kubectl get namespace "$NAMESPACE"

Expected result: the displayed context matches the rehearsal cluster, and the namespace is reported as active. If the context is wrong, stop here. Do not rely on the namespace name alone to distinguish production from a test cluster.

Set the metadata database connection for the rehearsal deployment through the mechanism your team already uses. Confirm it points to the isolated database before installing or upgrading. A simple review of the chart values may not reveal a connection assembled from environment variables, mounted configuration, or an external secret. The operator should verify the effective target through the approved, non-secret configuration path.

For deployments with tenant-specific access rules, include a test account and a known pair of allowed and disallowed records in the rehearsal dataset. Row-level security is not the focus of this upgrade drill, but an upgrade is not validated merely because a login page loads. See production patterns for Superset row-level security for the separate design problem of defining and testing those boundaries.

3. Capture the baseline before changing anything

Start by recording the state that recovery will need to reproduce. Save the output of the release history, current chart values, and Kubernetes workload status in the change record. Treat rendered values as sensitive if they contain or could reveal credentials. Helm commands that display values may expose secrets depending on how the deployment is configured; use an access-controlled location and redact before sharing.

helm history "$RELEASE" --namespace "$NAMESPACE"
helm get values "$RELEASE" --namespace "$NAMESPACE" --all
kubectl get deployments,statefulsets,pods -n "$NAMESPACE" -o wide
kubectl get events -n "$NAMESPACE" --sort-by=.lastTimestamp

Expected result: release history identifies the current revision and status; workload listings show the expected pods; recent events do not reveal an unresolved rollout problem. If your rehearsal release is not installed yet, use the equivalent commands against the live release only for read-only discovery, and keep the test deployment separate.

Record more than a revision number. Save the exact chart package or chart version, the image reference, non-secret values, configuration revision, database backup identifier, and time of backup. Record whether any schema migration has already run. If a prior release changed the metadata database outside the Helm lifecycle, capture that fact too. The point is to give the recovery operator enough information to compare the application state with the database state instead of guessing from the latest pod status.

Take a metadata backup using your database team’s supported procedure. Then verify that the backup can be identified and restored to the isolated recovery database. A job marked complete is evidence that a backup operation finished, not proof that the resulting data can be restored. The drill should record the restore target and the validation result without copying sensitive data into a shared ticket.

Create a baseline query and a baseline access test before the upgrade. The query should use a small, stable dataset with known expected rows and totals. The access test should use two identities with deliberately different permissions. Write down the expected result for each identity before running the test; otherwise a changed result can be misread as normal variation.

4. Prepare the proposed upgrade without applying it

Review the target chart and application versions as one proposed release. The chart, image, configuration, and metadata migration path form a combined change. Changing several of them at once can make failures harder to isolate, so document which components are changing and why. If a chart update is required independently of the application update, consider separating the changes into distinct reviewed steps where that is compatible with your release process.

Render the target manifests locally using the intended values and the exact chart version. A representative command is:

helm template "$RELEASE" superset/superset \
  --version "$TARGET_CHART_VERSION" \
  --namespace "$NAMESPACE" \
  -f rehearsal-values.yaml > rendered-target.yaml

The repository name and chart reference above are examples; use the chart source configured by your organization. Expected result: Helm writes a rendered manifest to the named file and exits successfully. Inspect the output for unexpected image references, service account changes, storage changes, ingress exposure, and resource replacements. Rendering is not proof that a live upgrade will succeed, but it lets reviewers catch accidental changes before applying them.

Validate the values against the selected chart version and compare the rendered target with the current deployment. In particular, investigate removed or renamed values, changes to pod labels relied on by policies, and any resource that appears to be replaced rather than updated. Do not assume that a successful template render guarantees compatibility with the existing metadata schema.

Write down the intended recovery order before proceeding. A useful high-level sequence is: stop or contain new application traffic if necessary; determine whether migration began or completed; restore metadata to a recovery target when required; deploy an application and chart combination compatible with that restored metadata; then validate normal queries and access. Exact actions depend on your infrastructure and the migration state. Avoid publishing a generic command that restores over a database without identifying its target and impact.

flowchart TD
  accDescr: Workflow stages and decisions: Record baseline, Back up metadata, Apply target release, Check migration and workloads, Run query and access tests, Accept or stop release, Restore and redeploy if needed. The adjacent text explains the conditions and exceptions.
  accTitle: Superset on Kubernetes — Upgrade and Recovery Drills That Matter workflow
  A["Record baseline"] --> B["Back up metadata"]
  B --> C["Apply target release"]
  C --> D["Check migration and workloads"]
  D --> E["Run query and access tests"]
  E --> F["Accept or stop release"]
  D --> G["Restore and redeploy if needed"]

The flow separates application health checks from data and permission checks. Reaching a running pod is an intermediate signal, not the acceptance decision. If migration or workload checks fail, stop the upgrade path and assess recovery; if the query or access tests fail, do not accept the release just because the user interface responds. The restore branch is not automatically triggered by every warning: the operator first determines the database state and uses the rehearsed recovery procedure appropriate to that state.

5. Apply the upgrade in the rehearsal environment

Before applying the target, capture the current Helm revision and confirm that the metadata backup is available. Use your reviewed values file and a version-pinned chart reference. A typical command is:

helm upgrade "$RELEASE" superset/superset \
  --version "$TARGET_CHART_VERSION" \
  --namespace "$NAMESPACE" \
  -f rehearsal-values.yaml \
  --wait

The command is illustrative, not a universal deployment prescription. Whether --wait is appropriate depends on the chart resources and your process; it does not certify that Superset’s business queries or access rules are correct. Do not include a password directly in the command. If the release is not already installed, use the installation path your team has validated rather than treating upgrade as a substitute for setup.

Expected result: Helm reports a successful release operation, and the deployment reaches its configured readiness conditions. If the command times out, do not immediately repeat it with different values. First inspect release history, the rendered or recorded manifest, pod events, and migration logs to determine how far the change progressed.

Watch rollout status and the pods associated with the release:

kubectl rollout status deployment/REPLACE_WITH_WEB_DEPLOYMENT \
  -n "$NAMESPACE" --timeout=5m
kubectl get pods -n "$NAMESPACE" -o wide
kubectl get events -n "$NAMESPACE" --sort-by=.lastTimestamp

Replace the deployment name with the actual resource name in your rendered manifests. Expected result: the web deployment reports completion, pods become ready, and no new unresolved scheduling, image-pull, or configuration errors appear. If the chart runs separate workers, check those workloads too; a healthy web deployment does not establish that asynchronous tasks can run.

Check the migration outcome using the release’s documented operational procedure and available application logs. Record whether it completed, failed, or is still in progress. Do not infer migration success from a successful Helm command alone. If migration is still running, avoid starting a second upgrade or initiating rollback until the team understands whether the first migration process remains active.

When available through your established access path, verify that a user can reach the Superset interface and that a representative dashboard or chart loads. This is a smoke test, not the release decision. A page can load while a warehouse query fails, a cached result hides an error, or a user sees data they should not see.

6. Run the query and access acceptance checks

Use a small test table with an explicit expected result. For example, create a rehearsal dataset with columns account_id, region, event_date, and net_amount. Insert four rows: two for account A in the West region with amounts 10 and 15, and two for account B in the East region with amounts 20 and 30. For the date range containing all four rows, an unrestricted aggregate should return four rows and a total of 75. These are illustrative values for a test fixture, not a claim about a production workload.

Run the same saved query before and after the upgrade, with cache behavior controlled according to the team’s normal test method. Record the query identity, date range, expected row count, expected aggregate, observed result, and timestamp. If the result changes, rerun against the source database or inspect the generated query through the established process before deciding whether the difference comes from the application, data freshness, warehouse behavior, or caching.

Then test access using two identities. The account A identity should see only account A rows and a total of 25; the account B identity should see only account B rows and a total of 50. An unrestricted test identity may see both accounts and a total of 75 if that is intended for the rehearsal. The test fails if either restricted user sees the other account’s rows, even when the overall dashboard appears healthy.

Keep the identity and expected-access mapping fixed between baseline and post-upgrade runs. If the users, roles, dataset, or filter rules changed at the same time as the upgrade, the test no longer isolates upgrade behavior. For product teams building customer-facing reporting, access design deserves its own treatment; this walkthrough does not replace the separate work involved in a semantic layer for agents or customer-specific authorization design.

A practical acceptance record can look like this:

This table is a decision aid, not a substitute for evidence retention controls. Store detailed logs and query results where authorized operators can review them, and keep the publication or ticket summary free of credentials and unnecessary customer data.

7. Rehearse recovery rather than relying on a Helm rollback

A Helm rollback is useful for selecting an earlier release revision, but it does not undo a database migration or reverse independent changes to external services. That distinction is operationally important: an older application image may be incompatible with metadata that has already been migrated. Therefore, do not run rollback reflexively when a post-upgrade check fails.

First classify the failure. Did the chart fail before application pods started? Did pods start but fail readiness? Did the migration start, fail, or finish? Did the app become available but return incorrect query results? Did only a permission test fail? Preserve timestamps and relevant logs before changing state. The safest recovery action depends on those observations.

For a failure before migration, the recovery may be limited to restoring the prior release configuration, provided the metadata database is unchanged and the previous application combination remains compatible. In the isolated drill, test the team’s planned release-revision selection and confirm the resulting resources and image references. Treat this as a separate path from restoring metadata.

For a failure after a migration has changed metadata, a Helm rollback alone is not a complete recovery. In the rehearsal, restore the pre-upgrade backup to the isolated recovery database, deploy the compatible application and chart configuration against that database, then repeat the query and access checks. Keep the failed database copy intact if practical so operators can examine what happened without confusing it with the recovery target.

Use a recovery database name that is visually distinct from the failed and production databases. Have the operator verify the database endpoint before starting Superset. This is a simple control against the particularly damaging error of restoring the correct backup into the wrong destination. Record who verified the target and the method used, without placing credentials in the run record.

The drill should establish measurable timing without promising a universal recovery time. Record the time from the decision to recover until the web workload is ready, and separately record the time until both the known query and access tests pass. Those intervals answer different questions. A green workload can precede a usable analytics service, and a restored database can still reveal a permissions or configuration defect.

Do not declare success based solely on a successful restore command. Verify that expected dashboards or datasets are present, that the representative query returns its known values, and that both permitted and prohibited access cases behave as expected. A recovery that restores objects but changes tenant visibility is not a successful recovery for a multi-tenant service.

8. Worked rehearsal: a migration fails after new pods start

Consider a hypothetical mid-market analytics team preparing a routine Superset upgrade. Its rehearsal release uses a dedicated namespace, an isolated copy of metadata, and a small two-account fixture. The team records its current chart and image references, stores a pre-upgrade metadata backup identifier, and captures the baseline query outputs: four rows and a total of 75 for the unrestricted identity; 25 for account A; and 50 for account B.

The target release deploys new pods, but the migration check reports an error. The web deployment has some ready pods, and a browser can load the login page. A rushed operator might interpret those signals as sufficient to roll back the chart. The drill instead follows the prewritten sequence: preserve logs, stop additional rollout actions, classify the migration state, and keep the test isolated from production.

The team determines that the migration did not reach its expected completion state. It does not assume that the older application can safely use the changed metadata. The operator restores the identified pre-upgrade backup to a clearly named recovery database, confirms the effective connection target, and deploys the previously recorded compatible application and chart combination. The team then runs the three fixture queries and both account-specific access tests.

The recovery is accepted only if the data counts and totals match the baseline and the restricted identities remain isolated. If the unrestricted query passes but account A sees account B’s rows, the team marks the drill failed and investigates the restored configuration and authorization state. If the workload is ready but the query is stale, it checks the query path and cache behavior rather than treating readiness as proof of data accuracy.

The useful output of this exercise is not a claim that a future production incident will have the same cause or duration. It is a record of whether the team could identify the right backup, restore it to the intended destination, select a compatible application state, and establish service correctness with explicit tests. That record also exposes missing prerequisites, such as an unrecorded image digest or a backup that cannot be restored independently.

A counterexample helps define the boundary: suppose the release fails because a pod cannot pull its image and no migration was started. Restoring metadata would add risk without addressing the image problem. The operator should diagnose the image reference or access configuration and use the appropriate release recovery path. Recovery drills are valuable because they teach the team to distinguish failure modes, not because every failure should trigger the same backup procedure.

9. Troubleshooting common drill failures

Helm reports success, but pods are not ready. Inspect pod events, container states, image references, and configuration mounts. Compare the live resources with the saved rendered target. Check whether a required external dependency is reachable from the rehearsal environment. Do not repeatedly reapply the release until you know whether the issue is scheduling, image retrieval, application startup, or a dependency failure.

The rollout times out while migration status is unclear. Preserve the release history, events, and application logs. Determine whether a migration process is active and whether metadata has changed before retrying or selecting an earlier revision. If the state cannot be established, stop the drill and involve the database and application operators rather than making an irreversible assumption.

The prior Helm revision is available, but its values are incomplete. A revision reference does not replace the full recovery record. Compare its chart and image to the baseline, and check configuration and secret references that may have changed outside the release. If the prior application configuration cannot be reconstructed, the drill has found a documentation gap; do not improvise in production.

The restore completes, but Superset cannot connect. Verify that the application is configured for the recovery database, that the endpoint is reachable from the namespace, and that the appropriate secret version is available. Confirm that the recovery copy is ready for connections. Keep connection details out of shared logs while checking them through your approved process.

The dashboard loads, but the fixture total differs. Check the date range, source rows, query identity, and cache state. Confirm whether the query is running against the intended rehearsal database. Compare the underlying source result with the application result; do not accept a value simply because it resembles the previous result.

The aggregate is correct, but a restricted user sees extra rows. Treat this as a failed access test, not a cosmetic discrepancy. Record the identity and dataset used, stop release acceptance, and compare effective roles and filters with the baseline. Do not attempt to resolve a permission failure by broadening access just to make the dashboard appear functional.

A test passes once and fails on the next run. Check for changing fixture data, time-dependent filters, asynchronous refresh, and caches. Make the test deterministic by recording the fixture version, query definition, identity, and observation time. If variability is expected, define an explicit tolerance or stable subset before the next drill rather than explaining away a mismatch after it occurs.

10. Printable upgrade-and-recovery worksheet

Use this worksheet in the change record. Complete it before the upgrade begins, then fill in observed results during the exercise. It is a printable in-article artifact, not a downloadable file.

Before the drill

• Environment: Record cluster context, namespace, release name, and metadata database identifier. Confirm the namespace and database are isolated from production.
• Version set: Record current and target chart versions, Superset image references, configuration revision, and any separately deployed worker version. Note who reviewed the rendered target.
• Backup: Record the pre-upgrade backup identifier, creation time, intended restore target, and the operator who verified the target. Confirm that the backup can be restored without overwriting production.
• Baseline: Save release history, workload status, relevant events, migration state, and the known query and access-test outputs. Store sensitive details only in the approved location.
• Stop conditions: Name the person who can stop the drill and list the conditions that pause execution, including an unknown migration state or an unexpected database endpoint.

During the drill

• Preflight: Confirm the active context, release namespace, effective database target, chart version, image reference, and rendered manifests.
• Upgrade: Record the command or change identifier, start time, Helm result, and resulting release revision. Do not put credentials in the record.
• Migration: Classify the outcome as completed, failed, still running, or unknown. Attach or reference the authorized logs used to make that classification.
• Workloads: Record readiness for web and worker workloads, plus material events. Note whether the interface responds, but do not treat that alone as acceptance.
• Data and access: Record the expected and observed fixture row counts and totals for each test identity. Mark any cross-account visibility as a failure.

If recovery is required

• Failure classification: Record what failed and whether metadata migration began or completed. Preserve relevant logs before changing state.
• Recovery target: Identify the backup and isolated database that will receive it. Have a second operator verify the destination if that is part of your team’s process.
• Compatible release: Record the chart, application image, and configuration selected for the restored metadata. Confirm the effective database target before starting the application.
• Verification: Rerun workload checks, the known query, and both allowed and denied access tests. Record the time to workload readiness separately from the time to correct results.
• Disposition: Mark the drill accepted, failed, or incomplete. List missing artifacts and assign an owner and review date for each gap.

The worksheet should make the next decision easier: whether the team has enough evidence to accept the release, whether a failure requires metadata restoration, or whether the drill must stop because the state is unknown. It should not collapse those decisions into a single “pods green” checkbox.

11. Turn the drill into an upgrade gate

After the rehearsal, review every difference between expected and observed results. A successful drill does not prove that every future migration will be recoverable; it demonstrates that the team exercised a defined path with a specific version set, backup, and test fixture. Keep that scope visible when translating the result into a release decision.

Use findings to improve the next change record. If operators could not identify the effective database endpoint, make that information safely discoverable. If the backup could not be restored without assistance, define the handoff before the production window. If permissions were not tested, add stable test identities and expected results. If chart rendering changed more than expected, narrow the upgrade or improve review of the selected chart version.

A production change should proceed only when the release owner can name the current and target versions, identify the pre-change recovery point, explain the migration state checks, and state the data and access criteria for acceptance. For teams comparing analytics products as their deployment needs change, the Superset and Metabase decision framework addresses that product choice separately from this operational upgrade procedure.

If the upgrade path also supports a customer-facing analytics experience, have the team review the complete deployment and access design before release. PADISO’s embedded analytics service is one appropriate next step for teams that need implementation help with that broader product integration.

An upgrade drill matters when it answers a concrete question with evidence: can this team return to a known-compatible application and metadata state, then demonstrate that queries are accurate and users see only the records they should? If the answer is not yet clear, the exercise has still done its job by revealing what must be fixed before the next production change.

WANT TO TALK THROUGH YOUR SITUATION?

Talk it through with us. Thirty minutes, no slides.