Skip to main content

Tableau and the data.world Collector

Introduction

Note

The latest version of the Collector is 2.129. To view the release notes for this version and all previous versions, please go here.

The data.world Collector harvests metadata from your source system. Please read over the data.world Collector FAQ to familiarize yourself with the Collector.

Prerequisites

  • You must have a ddw-catalogs (or other) dataset set up to hold your catalog files when you are done running the collector.

  • The machine running the catalog collector should have connectivity to the internet or access to the source instance. It is recommended to have a minimum of 8GB memory and a 2Ghz processor.

  • Docker must be installed. For more information see https://docs.docker.com/get-docker/.

  • The user defined to run the data.world Collector must have read access to all resources being cataloged.

  • The computer running the data.world Collector needs a Java Runtime Environment. OpenJDK 17 is supported and available here.

Version of data source

This is the version of the underlying metadata sources that we have developed/tested against. This is not to say that the collector will not function properly with other versions, but this is the version we specifically developed against:

Permissions required to catalog Tableau

  • Site Administrator (Tableau Online site or Tableau Server admin)

  • If derived permissions is enabled by your Tableau Online site admin or Tableau Server admin, then your Tableau User with Creator or Explorer license can see metadata on external assets for the content that you own, or for the content for which you are a project leader or project owner. You can also see metadata on external assets to which you have been granted explicit View permissions.

  • The collector harvests metadata only using the Tableau APIs. The collector does not require access to Tableau’s underlying storage (i.e., PostgreSQL)

What is cataloged

The information cataloged by the collector includes:

Lineage Tableau

The Tableau collector identifies:·      

  • The fields from which database columns and tables source their data

  • The fields from which sheets and dashboards source their data

  • The tables from which Tableau Views source their data

  • The columns and tables from which Custom SQL Tables source their data

Ways to run the data.world Collector

There are a few different ways to run the data.world Collector--any of which can be combined with an automation strategy to keep your catalog up to date:

  • Create a configuration file (config.yml) - This option stores all the information needed to catalog your data sources. It is an especially valuable option if you have multiple data sources to catalog as you don't need to run multiple scripts or CLI commands separately.

  • Run the collector though a CLI - Repeat runs of the collector requires you to re-enter the command for each run.

Note

This section walks you through the process of running the collector using CLI.

Writing the data.world Collector command

The easiest way to create your Collector command is to:

  1. Copy the following example command

  2. Edit it for your organization and data source

  3. Open a terminal window in any Unix environment that uses a Bash shell and paste your command into it.

The example command includes the minimal parameters required to run the collector (described below)--your instance may require more. A description of all the available parameters is available in this article. Edit the command by adding any other parameters you wish to use, and by replacing the values for all your parameters with your information as appropriate. Parameters required by the Collector are in bold.

Tableau and SAML

When SAML is enabled for Tableau, you must use Personal Access Tokens (PATs), not username-and-password-based authentication, to use the Tableau API (and therefore also the data.world Collector). Also note that if you use the same PAT to do multiple authentications simultaneously, you may see race conditions. So we recommend you create a dedicated PAT for use with the data.world Collector. As the majority of the people who want to catalog Tableau with the data.world Collector use SAML, the example CLI command below use a PAT instead of username and password. Further information about Tableau PAT can be found here for Tableau Online and here for Tableau Server.

Important

Do not forget to replace x.y in datadotworld/dwcc:x.y with the version of the Collector you want to use (e.g., datadotworld/dwcc:2.113).

Basic parameters

Each collector has parameters that are required, parameters that are recommended, and parameters that are completely optional. Required parameters must be present for the command to run. Recommended parameters are either:

  • parameters that exist in pairs, and one or the other must be present for the command to run (e.g., --agent and --base)

  • parameters that we recommend to improve your experience running the command in some way

Together, the required and recommended parameters make up the Basic parameters for each collector. The Basic parameters for this collector are:

--tableau-exclude option

When cataloging the metadata from a Tableau server you have the ability to exclude object types that you don't want cataloged. The option to use is --tableau-exclude, and it allows exclusion of the following 12 object types:

  • View

  • Dashboard

  • Database

  • PublishedDataSource

  • EmbeddedDataSource

  • CalculatedField

  • ColumnField

  • BinField

  • GroupField

  • DatasourceField

  • CustomSQLTable

  • Metric

Here is an example command with multiple object types excluded.:

docker run -it --rm --mount type=bind,source=/tmp,target=/dwcc-output \
--mount type=bind,source=/tmp,target=/app/log datadotworld/dwcc:x.y \
catalog-tableau --tableau-api-base-url <baseUrl> \
--tableau-password <password> --tableau-username <username> \
-a <account> -n <catalogName> -o "/dwcc-output"\
--tableau-exclude="EmbeddedDataSource" --tableau-exclude=\
"ColumnField" --tableau-exclude="Metric"

Docker and the data.world Collector

Detailed information about the Docker portion of the command can be found here. When you run the command, run will attempt to find the image locally, and if it doesn't find it, it will go to Dockerhub and download it automatically:

dwcc_and_cli.png

Handling custom certificates

If the target data instance has a custom SSL certificate, we recommend extending our Docker image and installing the custom cert like this (where ./ca.der is the name and location of the cert file).

Dockerfile:

FROM datadotworld/dwcc:<collector_version>
ADD ./ca.der ca.der 
RUN keytool -importcert -alias startssl -cacerts -storepass changeit -noprompt -file ca.der

Important

Do not forget to replace <collector_version> in datadotworld/dwcc:<collector_version> with the version of the Collector you want to use (for example, datadotworld/dwcc:2.126).

Then, in the directory with that Dockerfile:

docker build -t dwcc-cert .

Note

The command needs to be all lower case, and there must be a dot (.) supplied to reference the current directory.

Finally, change the docker run command to use dwcc-cert instead of dwcc. Here is an example command for cataloging Tableau with a custom SSL certificate:

docker run -it --rm --mount type=bind,source=/tmp,target=/dwcc-output \
--mount type=bind,source=/tmp,target=/app/log dwcc-cert \
catalog-tableau --tableau-api-base-url <baseUrl> \
--tableau-password <password> --tableau-username <username> \
-a <account> -n <catalogName> -o "/dwcc-output"

Collector runtime and troubleshooting

The catalog collector may run in several seconds to many minutes depending on the size and complexity of the system being crawled. If the catalog collector runs without issues, you should see no output on the terminal, but a new file that matching *.dwec.ttl should be in the directory you specified for the output. If there was an issue connecting or running the catalog collector, there will be either a stack trace or a *.log file. Both of those can be sent to support to investigate if the errors are not clear. A list of common issues and problems encountered when running the collectors is available here.

Upload the .ttl file generated from running the Collector

When the data.world Collector runs successfully, it creates a .ttl file in the directory you specified as the dwcc-output directory. The automatically-generated file name is databaseName.catalogName.dwec.ttl. You can rename the file or leave the default, and then upload it to your ddw-catalogs dataset (or wherever you store your catalogs).

Caution

If there is already a .ttl catalog file with the same name in your ddw-catalogs dataset, when you add the new one it will overwrite the existing one.

Automating updates to your metadata catalog

Keep your metadata catalog up to date using cron, your Docker container, or your automation tool of choice to run the catalog collector on a regular basis. Considerations for how often to schedule include:

  • Frequency of changes to the schema

  • Business criticality of up-to-date data

For organizations with schemas that change often and where surfacing the latest data is business critical, daily may be appropriate. For those with schemas that do not change often and which are less critical, weekly or even monthly may make sense. Consult your data.world representative for more tailored recommendations on how best to optimize your catalog collector processes.

In this section: