Skip to content

Job Execution

dagctl executes jobs differently depending on the framework. Both use Kubernetes Jobs triggered by CronJobs, but the execution model differs.

SQLMesh Execution

Architecture

Each SQLMesh project gets its own dedicated set of resources in the organization namespace, including a sqlmesh-api deployment, a git-sync deployment, and a shared workspace volume. When a CronJob fires:

  1. Kubernetes creates a Job with dagctl annotations
  2. Kyverno intercepts pod creation and injects:
    • Environment variables from ConfigMaps and Secrets
    • State database credentials
    • Resource limits
  3. The SQLMesh runner drains the project's work queue
  4. Job Watcher detects completion, parses logs, and reports status

Concurrency Model

The Work Queue

SQLMesh execution is driven by a durable work queue rather than by the command configured on the job. A separate queue-populate CronJob runs every 5 minutes, checks the full model graph for models with intervals due, and enqueues a work item for each one.

When a job run starts, its pod becomes a worker that:

  1. Polls the queue for the next available work item. A model only becomes available once all of its upstream dependencies have completed, so execution always respects the DAG. Among models that are simultaneously ready, the deepest ones are preferred so the longest downstream chains are unblocked soonest.
  2. Heartbeats while the model executes, holding a visibility timeout on that item
  3. Marks the item complete and polls for the next one, until the queue is drained

Because items are held by a visibility timeout rather than an in-memory claim, work is never lost when a pod dies — an item whose worker stops heartbeating becomes visible again and is retried by the next worker, up to 3 attempts.

Concurrent Job Runs

Overlapping runs are rejected by default

max_concurrent_job_runs defaults to 1. If a scheduled run starts while a previous run of the same project is still executing, the new run is rejected at registration and reports a status of skipped — it does not queue up and it does not execute later.

If you schedule a job more frequently than it takes to run, you will silently lose runs. Either raise max_concurrent_job_runs on the project, or set the cron interval comfortably above your typical run duration.

Parallel Model Execution (Within a Job)

Within a single job run, dagctl executes multiple models concurrently while respecting their dependencies:

  1. Models are analyzed for their upstream dependencies at the start of execution
  2. Ready models execute immediately — Models with no unmet dependencies start right away
  3. Execution is limited by the MAX_CONCURRENT_MODELS setting (default: 5 models at once)
  4. As models complete, new ready models are immediately submitted for execution
  5. Failed dependencies block downstream models — If a model fails, models that depend on it are marked as blocked

Configuration

Control model-level concurrency using the MAX_CONCURRENT_MODELS environment variable:

  • Default: 5 concurrent models
  • Recommendation: Start with 5 and increase based on your infrastructure capacity
  • Considerations: Higher values increase parallelism but require more memory and database connections

Edge Cases

Environment Changes During Execution

If your SQLMesh environment is modified during a job run (e.g., a new plan is applied):

  1. Currently running models complete gracefully
  2. New models will not start — the job stops pulling additional work items
  3. Unprocessed work items remain in the queue
  4. Next scheduled run processes remaining work with the updated environment

Job Failures

  • Model-level failures — Individual model failures don't stop the job; other independent models continue
  • Dependency failures — Models that depend on failed models are marked as "blocked" and not executed
  • Infrastructure failures — Pod eviction or OOM releases the worker's in-flight work item back to the queue once its visibility timeout expires, and it is retried on a later run

dbt Execution

Architecture

Each dbt project has its own dedicated set of containers deployed in the organization namespace:

  • git-sync — Continuously tracks the repository for changes
  • dbt-api — Handles dbt command execution and build management
  • poller — Detects new deployable versions
  • job-watcher — Monitors execution and reports status back to the management API
  • sync-reporter — Reports repository sync state back to the management API

Image-Based Execution

dbt jobs run from a container image built at deploy time via Kaniko. This means:

  • ✅ Code changes require a new deploy or plan to take effect
  • ✅ Consistent execution — The same image runs for all job executions until redeployed
  • ✅ Build logs available in the Plans tab for debugging image issues

Job Status

Job Watcher monitors execution and reports completion status to the management API. For each run, it:

  1. Watches the Kubernetes Job for completion or failure
  2. Parses run_results.json from pod logs for model-level execution details
  3. Reports individual model execution records (rows processed, duration, status)
  4. Updates the run status in the web UI

Key Takeaways (Both Frameworks)

  • ✅ Jobs run on Kubernetes — No persistent workers to manage
  • ✅ Secrets are injected at runtime — Never stored in your repository
  • ✅ Job status and logs are always available in the web UI
  • ✅ Slack alerts fire on failures for both frameworks

Next Steps