Preparing to run the dbt cloud collector
Pre-requisites for running the collector
Item | Requirement |
|---|---|
Hardware | |
RAM | 8 GB |
CPU | 2 Ghz processor |
Software | |
Docker | Click here to get Docker. |
Java Runtime Environment | OpenJDK 17 is supported and available here. |
data.world specific objects | |
Dataset | You must have a ddw-catalogs (or other) dataset set up to hold your catalog files when you are done running the collector. |
Preparing dbt cloud for collectors
Obtaining account ID, project ID, Job ID, and job run ID
This section talks about generating the account ID, project ID, job ID, and job run ID. You will use this information in the collector command/YAML file.
The dbt cloud collector assumes that your dbt Cloud instance has an environment and job set up with at least one successful run.
To obtain this information:
Under the Deploy menu at the top navigation, go to Jobs.
Copy the Account ID and the Project ID from the URL:
https://cloud.getdbt.com/deploy/<accountID>/projects<projectID>/jobs
Under the Deploy menu, go to Environment, and select the environment that you want to run the collector against.
You can identify the Environment Name from the title on this page, or the Environment ID from the URL:
https://cloud.getdbt.com/deploy/<accountID>/projects<projectID>/environments/<environmentID>
From the Jobs section, select the Job associated with your Environment
You can identify the Job name from the title on this page, or the Job ID from the following URL:
https://cloud.getdbt.com/deploy/<accountID>/projects<projectID>/jobs/<jobID>
If you want the collector to harvest from a specific dbt Cloud job run, and not the most recent successful job run, then from the Job page, click on a specific job run. Copy the Job Run ID from the URL:
https://cloud.getdbt.com/deploy/<accountID>/projects<projectID>/runs/<jobRunID>.
Obtaining dbt Cloud API token
From the top right menu dropdown, select Profile Settings.
Navigate to the API section. Click copy to the right of the API Key. You will use this api key for --dbt-cloud-api-key when setting up the collector to authenticate to dbt cloud.
Updating job execution settings
Under the Deploy menu at the top navigation, go to Jobs.
From the Environment dropdown, select the environment that you want to run the collector against.
Select the Job associated with your Environment and click Settings.
Under Execution Settings, ensure that Generate docs on run is selected.