Skip to content

Job Execution

dagctl executes jobs differently depending on the framework. Both use Kubernetes Jobs triggered by CronJobs, but the execution model differs.

SQLMesh Execution

Architecture

SQLMesh runs on a shared service. When a CronJob fires:

  1. Kubernetes creates a Job with dagctl annotations
  2. Kyverno intercepts pod creation and injects:
    • git-sync init container (clones your repository)
    • Environment variables from ConfigMaps and Secrets
    • State database credentials
    • Resource limits
  3. The SQLMesh runner executes the configured command
  4. Job Watcher detects completion, parses logs, and reports status

Concurrency Model

Parallel Jobs (Model Claiming)

When multiple job instances run simultaneously — such as when a scheduled job starts before the previous run completes — dagctl uses model claiming to prevent duplicate work:

  1. Before execution begins, each job requests to "claim" the models it needs to run
  2. If a model is already claimed by another running job, that model is skipped
  3. As each model completes, its claim is immediately released, making it available for other jobs

This ensures:

  • ✅ No duplicate execution — The same model never runs twice simultaneously across different jobs
  • ✅ Efficient resource usage — Models are processed as soon as they're available
  • ✅ Safe overlapping runs — You can schedule jobs frequently without worrying about conflicts

Parallel Model Execution (Within a Job)

Within a single job run, dagctl executes multiple models concurrently while respecting their dependencies:

  1. Models are analyzed for their upstream dependencies at the start of execution
  2. Ready models execute immediately — Models with no unmet dependencies start right away
  3. Execution is limited by the MAX_CONCURRENT_MODELS setting (default: 5 models at once)
  4. As models complete, new ready models are immediately submitted for execution
  5. Failed dependencies block downstream models — If a model fails, models that depend on it are marked as blocked

Configuration

Control model-level concurrency using the MAX_CONCURRENT_MODELS environment variable:

  • Default: 5 concurrent models
  • Recommendation: Start with 5 and increase based on your infrastructure capacity
  • Considerations: Higher values increase parallelism but require more memory and database connections

Edge Cases

Environment Changes During Execution

If your SQLMesh environment is modified during a job run (e.g., a new plan is applied):

  1. Currently running models complete gracefully
  2. New models will not start — the job stops submitting additional models
  3. Claims are released so the next run can pick them up
  4. Next scheduled run processes remaining work with the updated environment

Job Failures

  • Model-level failures — Individual model failures don't stop the job; other independent models continue
  • Dependency failures — Models that depend on failed models are marked as "blocked" and not executed
  • Infrastructure failures — Pod eviction or OOM will leave some models unclaimed for the next run

dbt Execution

Architecture

Each dbt project has its own dedicated set of containers deployed in the organization namespace:

  • git-sync — Continuously tracks the repository for changes
  • dbt-api — Handles dbt command execution and build management
  • poller — Detects version changes in .dagctl/config.yaml
  • job-watcher — Monitors execution and reports status back to the management API

Image-Based Execution

dbt jobs run from a container image built at deploy time via Kaniko. This means:

  • ✅ Code changes require a new deploy or plan to take effect
  • ✅ Consistent execution — The same image runs for all job executions until redeployed
  • ✅ Build logs available in the Plans tab for debugging image issues

Job Status

Job Watcher monitors execution and reports completion status to the management API. For each run, it:

  1. Watches the Kubernetes Job for completion or failure
  2. Parses run_results.json from pod logs for model-level execution details
  3. Reports individual model execution records (rows processed, duration, status)
  4. Updates the run status in the web UI

Key Takeaways (Both Frameworks)

  • ✅ Jobs run on Kubernetes — No persistent workers to manage
  • ✅ Secrets are injected at runtime — Never stored in your repository
  • ✅ Job status and logs are always available in the web UI
  • ✅ Slack alerts fire on failures for both frameworks

Next Steps