Version:

Async Queries via Celery

Celery

On large analytic databases, it’s common to run queries that execute for minutes or hours. To enable support for long running queries that execute beyond the typical web request’s timeout (30-60 seconds), it is necessary to configure an asynchronous backend for Superset which consists of:

one or many Superset workers (which is implemented as a Celery worker), and can be started with the celery worker command, run celery worker --help to view the related options.
a celery broker (message queue) for which we recommend using Redis or RabbitMQ
a results backend that defines where the worker will persist the query results

Configuring Celery requires defining a CELERY_CONFIG in your superset_config.py. Both the worker and web server processes should have the same configuration.

class CeleryConfig(object):
    broker_url = "redis://localhost:6379/0"
    imports = (
        "superset.sql_lab",
        "superset.tasks.scheduler",
    )
    result_backend = "redis://localhost:6379/0"
    worker_prefetch_multiplier = 10
    task_acks_late = True
    task_annotations = {
        "sql_lab.get_sql_results": {
            "rate_limit": "100/s",
        },
    }

CELERY_CONFIG = CeleryConfig

To start a Celery worker to leverage the configuration, run the following command:

celery --app=superset.tasks.celery_app:app worker --pool=prefork -O fair -c 4

To start a job which schedules periodic background jobs, run the following command:

celery --app=superset.tasks.celery_app:app beat

To setup a result backend, you need to pass an instance of a derivative of BaseCache (from flask_caching.backends.base import BaseCache) to the RESULTS_BACKEND configuration key in your superset_config.py. You can use Memcached, Redis, S3, MinIO, memory or the file system (in a single server-type setup or for testing), or to write your own caching interface. Your superset_config.py may look something like:

# On S3
from s3cache.s3cache import S3Cache
S3_CACHE_BUCKET = 'foobar-superset'
S3_CACHE_KEY_PREFIX = 'sql_lab_result'
RESULTS_BACKEND = S3Cache(S3_CACHE_BUCKET, S3_CACHE_KEY_PREFIX)

# On Redis
from flask_caching.backends.rediscache import RedisCache
RESULTS_BACKEND = RedisCache(
    host='localhost', port=6379, key_prefix='superset_results')

For performance gains, MessagePack and PyArrow are now used for results serialization. This can be disabled by setting RESULTS_BACKEND_USE_MSGPACK = False in your superset_config.py, should any issues arise. Please clear your existing results cache store when upgrading an existing environment.

Important Notes

It is important that all the worker nodes and web servers in the Superset cluster share a common metadata database. This means that SQLite will not work in this context since it has limited support for concurrency and typically lives on the local file system.
There should only be one instance of celery beat running in your entire setup. If not, background jobs can get scheduled multiple times resulting in weird behaviors like duplicate delivery of reports, higher than expected load / traffic etc.
SQL Lab will only run your queries asynchronously if you enable Asynchronous Query Execution in your database settings (Sources > Databases > Edit record).
In order to use dedicated results backend, additional python libraries must be installed. These libraries can be installed using pip.
- redis-py for Redis.
- pylibmc for memcached
- s3werkzeugcache for S3
- minio-flask-cache for MinIO or other S3 compatible service

Celery Flower

Flower is a web based tool for monitoring the Celery cluster which you can install from pip:

pip install flower

You can run flower using:

celery --app=superset.tasks.celery_app:app flower

Additional Resources

Blog: How to Set Up Global Async Queries (GAQ) in Apache Superset

Task infrastructure and runtime feature flags

Task APIs, views, TaskManager and MCP task tools are registered unconditionally at boot. Registration does not connect to Redis or start workers. Redis, a data cache, Celery and distributed coordination must still be configured for async charts. Use normal migrations and superset init to provision permissions during upgrades.

The effective GLOBAL_TASK_FRAMEWORK feature flag gates Tasks UI, new task admission, Task APIs (including polling and cancellation), and MCP task invocation. Authentication, RBAC and subscriber filtering remain unchanged; the flag is not a replacement for authorization. The menu is evaluated per request. Runtime flag changes do not register/remove routes, tools or permissions and need no app restart; Use the supported feature-flag callbacks for dynamic changes; editing static FEATURE_FLAGS configuration still requires reloading the process configuration. Reload browser pages to refresh frontend bootstrap flags.

GLOBAL_ASYNC_QUERIES additionally gates async chart eligibility. Existing request opt-in, full JSON format/type, cache, subscriber identity and Task-read permission requirements remain. Ineligible requests use synchronous execution. The stock feature manager derives effective GTF-on whenever GAQ is on, including supported dynamic callbacks; no manager behavior is changed.

Effective GTF	GAQ	Task admission/API/MCP	Tasks UI	Eligible charts
Off	Off	Blocked	Hidden	Synchronous
On	Off	Available subject to permissions	Visible subject to permissions	Synchronous
On	On	Available subject to permissions	Visible subject to permissions	Asynchronous
Off	On (custom resolver only)	Blocked	Hidden	Synchronous fallback

FEATURE_FLAGS = {
    "GLOBAL_TASK_FRAMEWORK": True,  # generic tasks
    "GLOBAL_ASYNC_QUERIES": False,  # enable for async charts; implies GTF
}

Disabling task access

The flag is not a worker kill switch. Already-admitted workers remain ungated and can finish, but turning effective GTF off blocks all user-facing Task APIs, including polling and cancellation, as well as MCP invocation. Stop new submissions at all producers and drain outstanding work before disabling if clients must keep polling/cancelling. GAQ-off stops new async chart eligibility, not generic task submissions. With the stock manager, GAQ must also be off to make effective GTF off. Keep workers and coordination services available until admitted work has drained.

Celery​

Celery Flower​