PROVIDER Β· The analytics engineering and data transformation standard
dbt
dbt is the analytics engineering standard, moving transformation inside the warehouse where SELECT models, ref() dependencies, built-in tests, and generated docs give warehouse changes version control and CI.
4.5pts
Pros
Cons
Product Information
Detailed Notes
Overview
dbt, the data build tool, was created in 2016 by Fishtown Analytics, now dbt Labs, extracting transformation from ETL into the warehouse itself, where analysts write models in SELECT statements while dbt resolves dependencies, orchestrates, tests, and documents, standardising the ELT pattern.
For Backend & Database work the value is engineering discipline for SQL, since ref() links models into automatic execution order and lineage, built-in tests cover uniqueness, nullability, and relationships with a documentation site, and version control plus CI let warehouse changes be reviewed and regression tested like application code.
The barriers are prerequisites and cost, because a warehouse such as Snowflake, BigQuery, or Redshift must already exist with loading handled while dbt only transforms, Jinja templates and macros take learning, dbt Cloud bills per developer seat growing expensive for large teams, and open source needs self-built scheduling.
Core Strengths
- ref() resolves dependencies into execution order with a lineage graph
- Built-in generic and custom tests act as data quality gates in CI
- Automatic documentation sites expose models, columns, and test results
- Supports mainstream warehouses including Snowflake, BigQuery, Redshift, and Databricks
Product Ecosystem
Models and layering
Staging, intermediate, and marts layers with materialisations such as view, table, and incremental controlling cost.
Tests and docs
schema.yml declares tests and descriptions, dbt test runs assertions, and dbt docs generates a site.
Scheduling and ecosystem
dbt Cloud offers hosted scheduling and CI while open source runs under Airflow, Dagster, or similar orchestrators.
Drawbacks
- Transformation only without extraction or loading, requiring an existing warehouse
- Jinja templates and macros need learning with readability suffering on complex logic
- dbt Cloud prices per developer seat, growing costly as teams expand
- Open source needs self-built scheduling and CI with more operational effort
Use Cases
- Warehouse modelling and layered governance (β β β β β ) - Clear layers with lineage
- Data quality gates (β β β β β ) - Built-in tests in CI
- Analytics team collaboration (β β β β ) - Engineered SQL with review
- Light script transformation without a warehouse (β ) - Prerequisites unmet
Pricing
| Plan | Price | Scope |
|---|---|---|
| dbt Core | $0 | Apache 2.0 with self-built scheduling |
| dbt Cloud developer seat | From about USD 50 monthly | Hosted scheduling, CI, and docs |
| dbt Cloud enterprise | Contract quoted | Governance, SSO, support |
dbt Core is free but needs self-built scheduling and CI with Airflow or GitHub Actions, while dbt Cloud from about USD 50 per seat monthly hosts scheduling, CI, and docs, usually beating self-built operations efficiency above five people.
FAQ
-
Can dbt replace ETL tools? Not entirely since it only transforms inside the warehouse, with extraction and loading still handled by Fivetran, Airbyte, or custom pipelines, adding engineering to the transform layer.
-
Is dbt Cloud required? No, dbt Core is open source and complete, while Cloud sells hosted scheduling, CI, and docs access, so small teams can start on Core with GitHub Actions.
-
What does it offer over stored procedures? Models are versioned, testable, and documented with automatic lineage, whereas procedure logic hides in the database and costs far more to review and test.