Enterprise docs

dbt metadata collector

Prerequisites dbt
  • The computer running the catalog collector should have connectivity to the internet or access to the source instance, a minimum of 2G memory, and a 2Ghz processor.

  • Docker must be installed. For more information see https://docs.docker.com/get-docker/. If you can't use docker, we have a java version available as well -- contact us for more details.

Installing the collector
  1. Request access to a download link from your data.world representative for the catalog collector. Once you receive the link, download the catalog collector Docker image (or programmatically download it with curl).

  2. Load the docker image into the local computer’s Docker environment:

    docker load -i dwdbt-X.Y.tar.gz

    where X.Y is the version number of the dbt collector image.

  3. The previous command will return an <image id> which needs to be renamed as 'dwbt'. Copy the <image id> and use it in the docker-load command:

    docker tag <image id> dwdbt

A list of the parameter options that can be used with the dbt collector is at the end of this article. Where available, either short (e.g., -a) or long (--account) forms can be used.

Run the collector - dbt

catalog connector runtime

The catalog collector may run in several seconds to many minutes depending on the size and complexity of the system being crawled.

Automatic updates to your metadata catalog

Keep your metadata catalog up to date using cron, your Docker container, or your automation tool of choice to run the catalog collector on a regular basis. Considerations for how often to schedule include:

  • Frequency of changes to the schema

  • Business criticality of up-to-date data

For organizations with schemas that change often and where surfacing the latest data is business critical, daily may be appropriate. For those with schemas that do not change often and which are less critical, weekly or even monthly may make sense. Consult your data.world representative for more tailored recommendations on how best to optimize your catalog collector processes.