Data Engineering

Claude Code: /plugin marketplace add wshobson/agents → /plugin install data-engineering@claude-code-workflows

The Airflow material starts from four properties a DAG has to hold — idempotent, atomic, incremental, observable — and the retry defaults that follow from them, including exponential backoff with a capped maximum delay. Spark tuning goes past generic advice into the failure that actually stalls jobs: data skew, handled with salting, broadcast joins and adaptive query execution rather than by raising executor memory. Quality checks are mapped dimension by dimension onto concrete Great Expectations assertions and dbt tests, so completeness or timeliness becomes a named check in CI instead of a slide. Two agents and two commands cover the surrounding work, from warehouse design to shipping a feature that depends on a new pipeline.