Skip to content

SQLMesh Jobs

Quick Summary

SQLMesh jobs in dagctl are commands scheduled on a cron expression. They are the primary execution mechanism for keeping your SQLMesh environments up to date. Each job runs inside a Kubernetes pod that is injected with your project's credentials and state connection.

Setting Recommended Why
Job Type sqlmesh run + separate sqlmesh janitor Prevents janitor failures from blocking data execution
Schedule (Cron) */15 * * * * Matches most common model cadences
Slack Alerts Enabled on failures Get notified when jobs fail

Key Takeaways

  • ✅ Run sqlmesh run and sqlmesh janitor as separate jobs - A failed janitor should not disrupt production model execution
  • ✅ Schedule at the rate of your most frequent model - No more frequent than every 5 minutes
  • ✅ 15-minute intervals work well for most teams - Adjust only if your models require tighter SLAs

Job Types

sqlmesh run

Executes your SQLMesh models for the current interval. This is the primary job type. dagctl submits the job on schedule, and the execution pod runs sqlmesh run against your configured environment and state database.

# What dagctl runs inside the pod
sqlmesh run --environment prod

Models execute concurrently within the job, up to the MAX_CONCURRENT_MODELS limit. See Execution for details on the concurrency model.

sqlmesh janitor

Cleans up expired intervals and stale state entries. Janitor runs should be scheduled separately from sqlmesh run jobs.

# What dagctl runs inside the pod
sqlmesh janitor

Warning

Do not combine sqlmesh run and sqlmesh janitor in the same job. If the janitor fails, it will prevent model execution from completing. Run them on independent schedules so failures are isolated.

Scheduling

The cron schedule controls how often dagctl triggers a new job pod. Set the schedule to match the cadence of your most frequently-running model.

Model cadence Recommended job schedule
Hourly 0 * * * *
Every 30 minutes */30 * * * *
Every 15 minutes */15 * * * *
Every 5 minutes */5 * * * * (minimum)

dagctl uses model claiming to handle overlapping job runs safely. You can schedule jobs more frequently than their execution time without risking duplicate processing. See Execution for how claiming works.

Auto-Generate Jobs

dagctl can analyze your model DAG and suggest an optimized job configuration. Instead of running all models in a single sqlmesh run, auto-generate splits your DAG into parallel job groups based on model dependencies and cron schedules.

To use it:

  1. Open your project in the web UI
  2. Navigate to Jobs > Auto-Generate
  3. Review the suggested job topology
  4. Accept or modify before saving

Auto-generated configurations are a starting point. Review the suggested cron schedules against your actual model cron definitions before enabling them in production.

Warning

Auto-generate reads your Git repository at the current branch HEAD. If your model definitions are out of sync with what is deployed, the suggestions may not reflect your actual DAG. Sync your project before running auto-generate.

Concurrency

Within a single job run, dagctl executes models concurrently up to the MAX_CONCURRENT_MODELS limit (default: 5). You can override this per job using the environment variable setting on the job configuration.

For full details on how parallel execution and model claiming work, see Execution.

Next Steps