Running the SQL Server Reporting Services (SSRS) collector
Note
The latest version of the Collector is 2.159. To view the release notes for this version and all previous versions, please go here.
Ways to run the data.world Collector
There are a few different ways to run the data.world Collector--any of which can be combined with an automation strategy to keep your catalog up to date:
Create a configuration file (config.yml) - This option stores all the information needed to catalog your data sources. It is an especially valuable option if you have multiple data sources to catalog as you don't need to run multiple scripts or CLI commands separately.
Run the collector though a CLI - Repeat runs of the collector requires you to re-enter the command for each run.
Note
This section walks you through the process of running the collector using CLI.
Preparing and running the command
The easiest way to create your Collector command is to:
Copy the following example command in a text editor.
Set the required parameters in the command. The example command includes the minimal parameters required to run the collector
Open a terminal window in any Unix environment that uses a Bash shell and paste the command in it and run in.
docker run -it --rm --mount type=bind,source=${HOME}/dwcc,target=/dwcc-output \
--mount type=bind,source=${HOME}/dwcc,target=${HOME}/dwcc datadotworld/dwcc:2.144 \
catalog-ssrs --agent=8bank_org --output=/dwcc-output --api-token=${DW_AUTH_TOKEN} \
--no-log-upload=false --upload=true --name=bigorg-collection --upload-location=ddw-catalogs \
--username=8bank-user --password=${DW_SSRS_PASSWORD} \
--ssrs-api-base-url="http://ssrs-server.mycompany.com/api/3.10/
The following table describes the parameters for the command. Detailed information about the Docker portion of the command can be found here.
Parameter | Details | Required? |
|---|---|---|
dwcc:<CollectorVersion> | Replace | Yes |
-a =<agent> --agent=<agent> --account=<agent> | The ID for the data.world account into which you will load this catalog. The ID is the organization name as it appears in your organization. This is used to generate the namespace for any URIs generated. | Yes |
--ssrs-api-base-url=<baseUrl> | The Base URL of the SSRS API. For example, https://ssrs-server.mycompany.com/api/3.10 | Yes |
--username=<username> | SSRS username for authentication. | Yes |
--password=<password> | Password for the user. | Yes |
--pagination-limit=<paginationLimit> | The number of records per SSRS API page request. Default limit of per page is 200 records and max limit of per page is 2000 records. | No |
--dry-run=<dryRun> | Specify this option to run the collector in dry run mode to test the connection details provided. No metadata is harvested in dry run mode. | No |
-n=<catalogName> --name=<catalogName> | The name of the catalog - this will be used to generate the ID for the catalog as well as the filename into which the catalog file will be written. | Yes |
-o=<outputDir> --output=<outputDir> | The output directory into which any catalog files should be written. In our example we use the /dwcc -outputas it is running in a Docker container and that is what we specified in the script for a Docker mount point. You can change this value to anything you would like as long as it matches what you use in the mount point: -mount type=bind,source=/tmp,target=/dwcc-output ...-o /dwcc-output In this example, the output will be written to the /tmp directory on the local machine, as indicated by the mount point directive. The log file, in addition to any catalog files, will be written to the directory specified in the mount point directive. | Yes |
-L --no-log-upload | Do not upload the log of the dwcc run to the organization account's catalogs dataset or to another location specified with --upload-location (ignored if --upload not specified) | No |
--site=<site> | The slug for the data.world site into which you will load this catalog this is used to generate the namespace for any URIs generated. | No |
-H=Host --api-host=Host | The host for the data.world API. | No |
-t=<apiToken> --api-token=<apiToken> | The data.world API token to use for authentication. The default is to use an environment variable named DW_AUTH_TOKEN. | No |
-U --upload | Whether to upload the generated catalog to the organization account's catalogs dataset or to another location specified with --upload-location (This requires that the --api-token is specified. | No |
--upload-location=<uploadLocation> | The dataset to which the catalog is to be uploaded, specified as a simple dataset name to upload to that dataset within the organization's account, or [account/dataset] to upload to a dataset in some other account. This parameter is ignored if --upload is not specified. | No |
Common troubleshooting tasks
Collector runtime and troubleshooting
The catalog collector may run in several seconds to many minutes depending on the size and complexity of the system being crawled.
If the catalog collector runs without issues, you should see no output on the terminal, but a new file that matching *.dwec.ttl should be in the directory you specified for the output.
If there was an issue connecting or running the catalog collector, there will be either a stack trace or a *.log file. Both of those can be sent to support to investigate if the errors are not clear.
A list of common issues and problems encountered when running the collectors is available here.
Issue: The collector takes a long time to harvest metadata
Cause: The large size of an SSRS API results may take long time to harvest metadata. The collector paginates through the results which may lead to many API calls.
Solution: Increase the page size (--pagination-limit) to reduce the number of API calls.
Automating updates to your metadata catalog
Keep your metadata catalog up to date using cron, your Docker container, or your automation tool of choice to run the catalog collector on a regular basis. Considerations for how often to schedule include:
Frequency of changes to the schema
Business criticality of up-to-date data
For organizations with schemas that change often and where surfacing the latest data is business critical, daily may be appropriate. For those with schemas that do not change often and which are less critical, weekly or even monthly may make sense. Consult your data.world representative for more tailored recommendations on how best to optimize your catalog collector processes.
Overview
Some enterprise systems support the use of Secure Sockets Layer (SSL) encrypted communications on all external traffic. If you are harvesting metadata from a source system that requires SSL, you will need to add a CA certificate or self-signed certificate.
Obtaining the Custom SSL Certificate
Obtain the root certificate for your source system issued by your company. Typically your system administrator should be able to provide you with this.
Extending Docker to use custom SSL certificates
If the collector is run via Docker, extend the Docker image and install the custom certificate.
STEP 1: Prepare the Docker File
First, prepare a Dockerfile with the instructions for Docker to install the custom certificate and extend the Docker image.
Ensure you are on the machine where you have downloaded the Docker Image and plan to execute the Collector.
In a directory create the new Dockerfile with the following parameters for your custom SSL Certificate:
FROM datadotworld/dwcc:<collector_version> ADD ./<custom_certificate_file_path> <custom_certificate_file_name> RUN keytool -importcert -alias startssl -cacerts -storepass changeit -noprompt -file <custom_certificate_file_name>
Replace <collector_version> with the version of the Collector you want to use (For example, datadotworld/dwcc:2.120)
Replace <custom_certificate_file_path> with the path to the custom SSL Certificate.
Replace <custom_certificate_file_name> with the name of your custom SSL Certificate file.
For example, the command will look like:
FROM datadotworld/dwcc:2.120 ADD ./ ca.der certificate RUN keytool -importcert -alias startssl -cacerts -storepass changeit -noprompt -file certificate
STEP 2: Install the certificate and extend the docker image
Next, execute the the Dockerfile to install the certificate and extend the data.world Collector Docker Image.
Using your terminal of choice, ensure you are in the directory containing the Dockerfile created in step 1.
Next, create the new extended Docker image, called dwcc-cert in this example, by executing the following command:
docker build -t dwcc-cert .
Important things to note:
The command must be all lowercase.
The command must include the period (.) at the end, which directs Docker to use the local directory for the Dockerfile created above.
For the new image, the command uses the name dwcc-cert You can change the name if you want.
STEP 3: Run collector using the custom certificate
Finally, run the collector using the custom Certificate.
Get the standard docker run command for the Data Source you are collecting from.
Change the docker run command to use dwcc-cert image instead of dwcc image.
Sample command for Tableau.
docker run -it --rm --mount type=bind,source=/tmp,target=/dwcc-output \ --mount type=bind,source=/tmp,target=/app/log dwcc-cert \ catalog-tableau --tableau-api-base-url <baseUrl> \ --tableau-password <password> --tableau-username <username> \ -a <account> -n <catalogName> -o "/dwcc-output"
If you are using YAML file for running the collector, edit the command to use dwcc-cert image instead of dwcc image.
docker run -it --rm --mount type=bind,source=${HOME}/dwcc,target=/dwcc-output \ --mount type=bind,source=${HOME}/dwcc,target=${HOME}/dwcc -e DW_AUTH_TOKEN=${DW_AUTH_TOKEN} \ -e DW_TABLEAU_PASSWORD=${DW_TABLEAU_PASSWORD} dwcc-cert \ --config-file=/dwcc-output/config-tableau.yml
Adding custom SSL certificates when using jar
If the collector is run via jar, add the certificate to the JVM truststore.
From the terminal, navigate to the directory containing the certificate.
Run the following command to add the SSL certificate to the truststore:
keytool -importcert -alias startssl -cacerts -storepass changeit -noprompt -file <custom_certificate_file_path>
Replace <custom_certificate_file_path> with the path to the custom SSL Certificate.
For example, the command will look like:
keytool -importcert -alias startssl -cacerts -storepass changeit -noprompt -file ca.der
Finally, run the collector using the original jar file command. Note that this command does not need any modifications.
Troubleshooting SSL certificate issues
Issue
The following error occurs while running the collector:
Caused by: javax.ws.rs.ProcessingException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Description
There was an issue connecting to the source system using the SSL certificate.
Solution
Check to make sure the SSL certificate has not expired.
Ensure you have the correct SSL certificate for the source system.