Skip to main content

About the dbt cloud collector

The dbt cloud collector connects to your dbt cloud project, harvest dbt assets, and column-level lineage relationships from database views associated with dbt assets.

Note

The latest version of the Collector is 2.159. To view the release notes for this version and all previous versions, please go here.

How does the dbt cloud collector work?

dbt Cloud is an extract, transform, and load (ETL) platform that allows users to specify jobs that move and transform data between relational database tables and views in a target database. It makes rich metadata available about the processes that perform this movement and transformation of data.

In order to fully represent lineage relationships between the tables, the collector parses the definition of any database views created by dbt Cloud, connecting to the target database to obtain database metadata where necessary.

How does the collector know which dbt Cloud run to harvest from?

To make sure that the correct metadata is being harvested, the dbt Cloud collector needs to know the Project, Job, and Environment combination.

In dbt Cloud, a job is typically executed on a schedule. The dbt Cloud collector uses the following logic to determine the job run from which it will obtain the metadata artifacts to process:

  • The collector examines only dbt Cloud runs associated with the dbt Cloud project specified by the user.

  • Any unsuccessful runs and any runs that did not produce metadata artifacts are discarded.

  • If the user specifies a job (by identifier or name) or an environment (by identifier or name), then only runs from that job and/or environment are examined.

  • If multiple successful runs are found, then the collector harvests the metadata from the most recent run.

Running target database collector in addition to dbt Cloud Collector

The dbt Cloud collector only captures the lineage relationships but does not harvest any information about the database tables/views and columns involved in lineage relationships. In order to harvest information such as the table/view or column name, descriptions, data types, and table/view-column associations, you need to run the appropriate database collector (for example, the Snowflake collector) on the target database.

What is cataloged

The information cataloged by the collector includes metadata for the following dbt resources:

Table 1.

Object

Information cataloged

Analysis

Name, Description, Path, Root path, Package name, Unique ID, Alias, Meta, Raw SQL, Compiled SQL, Enabled, Materialized, Resource type

Model

Name, Description, Path, Root path, Package name, Unique ID, Alias, Meta, Raw SQL, Compiled SQL, Enabled, Materialized, Resource type

Project

Name, Project version

Snapshot

Name, Description, Path, Root path, Package name, Unique ID, Alias, Meta, Raw SQL, Compiled SQL, Enabled, Materialized, Resource type

Seed

Name, Description, Path, Root path, Package name, Unique ID, Alias, Meta, Raw SQL, Compiled SQL, Enabled, Materialized, Resource type

Source

Name, Description, Path, Root path, Package name, Unique ID, Alias, Meta, Raw SQL, Compiled SQL, Enabled, Source name, Resource type

Test

Name, Description, Path, Root path, Package name, Unique ID, Alias, Meta, Raw SQL, Compiled SQL, Enabled, Materialized, Resource type



Note: The collector also harvests the following information about the dbt Cloud resources. This information is available in the collector output and not presented in the UI. You can use this information for querying and Eureka Automations.

  • The run that produced the artifacts

  • The job that configured the run

  • The environment for the job (that specifies the target database)

  • The dbt Cloud project in which the job is defined

  • The dbt Cloud account

Relationship between objects

By default, the data.world catalog will include catalog pages for the resource types below. Each catalog page will have a relationship to other related resource types. Note that the catalog presentation and relationships are fully configurable, so these will list the default configuration.

Table 2.

Resource page

Relationship

Model

  • Project containing dbt model

  • Tests testing the integrity of model, dbt resources (test, seed, model, snapshot, source) that are upstream of model

  • dbt resources (Test, Seed, Model, Snapshot, Source) that are downstream of model

Project

dbt resources (Test, Seed, Model, Snapshot, Source) contained within project

Snapshot

  • Project containing dbt project

  • dbt resources (Test, Seed, Model, Source) that are upstream of snapshot

  • dbt resources (Test, Seed, Model, Source) that are downstream of snapshot

Seed

  • Project containing dbt project

  • dbt resources (Test, Seed, Model, Snapshot, Source) that are upstream of seed

  • dbt resources (Test, Seed, Model, Snapshot, Source) that are downstream of seed

Source

  • Project containing dbt project

  • dbt resources (Test, Seed, Model, Snapshot) that are downstream of seed

  • Database table that is the source of data for source

Test

  • Project containing dbt project

  • dbt model that has its integrity tested by this test



Lineage for dbt

Table 3.

Object

Lineage available

dbt model materialized as view

  • Referenced database tables and columns in dbt model materialized as view

dbt resource

  • dbt resources that are upstream and downstream (for example, seeds that are upstream of models, and tests that are downstream of models) of dbt resource.



We also collect column-level lineage for the following databases in the dbt collector:

  • Snowflake

Important

For Eureka Explorer, these harvested lineage relationships display from the page of theupstreamordownstreamresource from dbt. For example, you can see and access Eureka Explorer from a downstream Snowflake table resource page to see what upstream Snowflake table was transformed as a result of a view associated with a dbt model. The dbt resource will also appear in Eureka Explorer.

Supported versions of dbt

The collector supports the following dbt versions:

  • dbt 1.4

Authentication supported

  • The collector supports authenticating to dbt cloud using API key.

  • When the collector authenticates to Snowflake as the target database, the collector supports either:

    • Username and key pair authentication.

    • Username and password authentication