Skip to main content

Running the Teradata collector on-premise


The latest version of the Collector is 2.189. To view the release notes for this version and all previous versions, please go here.

Ways to run the Collector

There are a few different ways to run the Collector--any of which can be combined with an automation strategy to keep your catalog up to date:

  • Create a configuration file (config.yml)Using a YAML file to run a collector - This option stores all the information needed to catalog your data sources. It is an especially valuable option if you have multiple data sources to catalog as you don't need to run multiple scripts or CLI commands separately.

  • Run the collector through a CLI - Repeat runs of the collector requires you to re-enter the command for each run.


This section walks you through the process of running the collector using CLI.

Preparing and running the command

The easiest way to create your Collector command is to:

  1. Copy the following example command in a text editor.

  2. Set the required parameters in the command. The example command includes the minimal parameters required to run the collector

  3. Open a terminal window in any Unix environment that uses a Bash shell and paste the command in it and run in.

docker run -it --platform linux/amd64 --rm \
--mount type=bind,source="$HOME/dwcc-drivers",target="/usr/src/dwcc-config/lib" \
--mount type=bind,source="$HOME/dwcc/",target="/dwcc-output" \
--mount type=bind,source="$HOME/dwcc/",target="/app/log" \
datadotworld/dwcc:2.168 catalog-teradata \
--agent "collector-test" --name "catalog_teradata" \
--server ${TERADATA_SERVER} --port 1025 --username ${TERADATA_USER} --password P ${TERADATA_PASSWORD} \
--database ${SAMPLE_DATABASE} \
--output "/dwcc-output"

The following table describes the parameters for the command.

Table 1.




--mount type=bind,source= ${HOME}/dwcc-drivers,target="/usr/src/dwcc-config/lib"

The location where you placed the JDBC driver.



Replace <CollectorVersion> in with the version of the collector you want to use (For example, datadotworld/dwcc:2.166)


-a =<agent>



The ID for the account into which you will load this catalog. The ID is the organization name as it appears in your organization. This is used to generate the namespace for any URIs generated.




Specify the hostname of the database server to connect to.




If your provided server name is localhost, specify a friendly name to the environment in which your database server runs to help differentiate it from other environments.




The port of the database server (if not the default).




The name of the database to connect to.



When the --database parameter is not provided in the command/YAML file, the collector harvests metadata from all databases. If you want the collector run to not harvest specific databases, use the --exclude-database parameter and specify one or more regular expressions to indicate databases not to be cataloged. This parameter is ignored if the --database is specified.




A unique identifier for this database - will be used to generate the ID for the database (this is optional, you only need to provide this if the database name used for the connection is not sufficiently unique to completely identify the database)




The username to use to make the JDBC connection.




Specify this option to provide the password for the database on the command line.



JDBC driver properties to pass through to driver connection, as name=value. Separate the name=value pairs with a semicolon (;). For example, property1=value1;property2=value2 



Specify whether to skip harvesting of intra-database lineage metadata.



Enable harvesting of column statistics (i.e. data profiling).

Note: Activating the profiling feature may extend the running time of the collector. This is because the collector needs to read the table data to be able to gather metadata for profiling.



Specify whether to enable sampling and storage of sample values for string-valued columns.



Target for number of rows to sample from tables.



Specify this option to run the collector in dry run mode to test the connection details provided. No metadata is harvested in dry run mode.




The name of the catalog - this will be used to generate the ID for the catalog as well as the filename into which the catalog file will be written.




The output directory into which any catalog files should be written.

In our example we use the /dwcc -output as it is running in a Docker container and that is what we specified in the script for a Docker mount point.

You can change this value to anything you would like as long as it matches what you use in the mount point:

-mount type=bind,source=/tmp,target=/dwcc-output ...-o /dwcc-output

In this example, the output will be written to the /tmp directory on the local machine, as indicated by the mount point directive. The log file, in addition to any catalog files, will be written to the directory specified in the mount point directive.




Do not upload the log of the dwcc run to the organization account's catalogs dataset or to another location specified with --upload-location (ignored if --upload not specified)



The slug for the site into which you will load this catalog this is used to generate the namespace for any URIs generated.




The host for the API.




The API token to use for authentication. The default is to use an environment variable named DW_AUTH_TOKEN.




Whether to upload the generated catalog to the organization account's catalogs dataset or to another location specified with --upload-location (This requires that the --api-token is specified.)



The dataset to which the catalog is to be uploaded, specified as a simple dataset name to upload to that dataset within the organization's account, or [account/dataset] to upload to a dataset in some other account. This parameter is ignored if --upload is not specified.


Common troubleshooting tasks

Collector runtime and troubleshooting

The catalog collector may run in several seconds to many minutes depending on the size and complexity of the system being crawled.

  • If the catalog collector runs without issues, you should see no output on the terminal, but a new file that matching *.dwec.ttl should be in the directory you specified for the output.

  • If there was an issue connecting or running the catalog collector, there will be either a stack trace or a *.log file. Both of those can be sent to support to investigate if the errors are not clear.

A list of common issues and problems encountered when running the collectors is available here.

Automating updates to your metadata catalog

Maintaining an up-to-date metadata catalog is crucial and can be achieved by employing Azure Pipelines, CircleCI, or any automation tool of your preference to execute the catalog collector regularly.

There are two primary strategies for setting up the collector run times:

  • Scheduled: You can configure the collector according to the anticipated frequency of metadata changes in your data source and the business need to access updated metadata. It's necessary to account for the completion time of the collector run (which depends on the size of the source) and the time required to load the collector's output into your catalog. This could be for instance daily or weekly. We recommend scheduling the collector run during off-peak times for optimal performance.

  • Event-triggered: If you have set up automations that refresh the data in a source technology, you can set up the collector to execute whenever the upstream jobs are completed successfully. For example, if you're using Airflow, Github actions, dbt, etc., you can configure the collector to automatically run and keep your catalog updated following modifications to your data sources.