Catalog collector release notes

Amazon S3 collector now harvests resources up to and including the maximum count specified by the user or the 10,000 default limit.

dbt Cloud collector handles scenarios where dbt Cloud runs contain more than the current dbt Cloud limit of 1000 artifacts.

Databricks collector properly harvests column statistics from tables with no columns.

MS SQL Server collector properly handles case sensitivity of SQL keywords when parsing lineage.

The following two new collectors are now available:

Power BI and power BI Gov Collectors: The following parameters for these two collectors now support regular expressions: --exclude-workspace --include-workspace

The following JDBC collectors now test for connection status before executing queries. If a connection is closed by the database, the collector detects this condition and re-opens the connection.

Databricks collector now properly handles non-alphanumeric characters in object names.

Amazon S3 collector now allows users to filter by bucket with --include-bucket and --exclude-bucket options.

Kafka - Confluent Platform Collector now supports SASL/SCRAM-SHA-512 authentication. If you want to use this authentication, use the --kafka-cluster-sasl-type parameter while running the collector.

New parameters for API retries when API calls fail: The following collectors now support two new parameters ( --api-max-retries and --api-retry-delay) which allow users to specify the maximum number of times the collector will try to reconnect when the API call to the data source fails.

dbt Cloud Collector now properly selects the job that the user specified while running the collector.

The AWS Glue collector properly handles AWS Glue Catalog instances with more than 100 databases.

AWS Glue collector has improved logging to help with troubleshooting of access and permissions issues.

DBT Cloud collector: The following enhacements are made to the collector.

Snowflake and SQL Server Collectors now harvest column-level lineage from Stored Procedures.

Logging improvements:

A new collector is now available for SQL Server Reporting Services.

Databricks collector: Update the collector to retry after a pause when the Databricks API responds with too many requests.

All database collectors: Optimized the database collectors to reuse database connections where possible.

All collectors: Fixed an issue that prevented the user from passing command-line options containing spaces, when running the collectors using the docker container.

Databricks Collector The collector now allows users to use Personal Access Token without specifying username/password for authentication.

Power BI Gov collector: The --include-user-workspace parameter is removed from the collector CLI options.

Databricks Collector has a new option --workflow-exclude to exclude harvesting of jobs/workflows.

Power BI and Power BI Gov Collectors now support parameter values in Power BI expressions.

Tableau collector properly handles duplicate data sources when multiple filtered projects are specified.

Power BI and Power BI Gov collectors: The collectors have a new option --max-parseable-expression-length, which sets the maximum number of characters in a PowerBI expression that will be parsed for lineage metadata.

Power BI and ThoughtSpot collectors now refresh expired authentication tokens.

The SQL Server collector now properly handles missing SQL definition when harvesting stored procedures.

Tableau collector: The collector now harvests lineage relationships between embedded data sources and published data sources to reflect any such relationship that exists in Tableau.

Power BI collector: Improvements made to the collector to avoid hitting Power BI Admin API rate limits that prevented successful collection for certain large Power BI organizations.

Marquez collector: API authentication token [--marquez-api-key] is now a required parameter for the collector.

Fivetran collector: Fivetran API key (--fivetran-apikey) and Fivetran secret (--fivetran-apisecret) options are now required parameters for the collector.

The following two new collectors are now available:

Databricks collector: The collector now properly handles missing information returned by the Databricks APIs.

dbt Core and dbt Cloud Collectors: The collectors now use the description property for resources in a dbt manifest file to populate the description of associated catalog resources.

Power BI collector: The collector now properly handles unexpected source formats.

DB2 collector: The collector now support harvesting of column statistics and function and stored procedure information. For details about using the new parameters (--target-sample-size, --sample-string-values, --enable-column-statistics) for these features, see the DB2 collector documentation.

Redshift collector: The collector now properly distinguishes between user-defined functions and stored procedures when harvesting function and stored procedure metadata in the collector.

Tableau collector: Improved error messages and handling of missing Salesforce connection information within the Tableau collector.

Databricks collector: Fixed defects in the collector to accommodate invalid number formats and missing information returned by Databricks APIs in some cases.

Power BI collector: A new parameter --all-workspaces-and-apps is available for the Power BI collector which allows users to catalog all available data from the tenant using the admin API.

Databricks collector: Fixed an issue where the collector was terminating abnormally when it encountered a notebook that had no language specified for it.

Microsoft SQL Server collector: Fixed an issue with parsing the SQL for certain Views in Microsoft SQL Server that prevented harvesting of lineage.

Thoughtspot collector now harvests:

Databricks collector now harvests additional metadata for Databricks tables.

The Redshift, SQL Server, and PostgreSQL collectors now harvest:

Power BI, Looker, and Thoughtspot collectors: The resources cataloged by these collectors will now automatically include a link to the resource in the source system. This allows users to go from data.world to the associated URL for the same resource in the source system so the users do not have to manually find that resource in the source system.

dbt cloud collector allows the user to pass in a Snowflake role and Snowflake warehouse to override values found in the dbt cloud project configuration.

dbt core and dbt cloud collectors properly handle source meta config values that are objects rather than strings in the generated dbt manifest file.

SQL Server Collector properly disables lineage collection when the --disable-lineage-collection parameter is set.

Databricks collector includes additional checks for existence of and access to Unity Catalog.

SQL Server collector now harvests created date and modified date for tables and schemas, and harvests table size in bytes.

Snowflake collector now harvests table size in bytes.

Power BI collector allows setting the --azure-tenantid option when using username and password authentication.

All collectors now support the ability to set the JVM stack size using the DWCC_JVM_OPTIONS parameter.

SQL Server collector properly handles harvesting of View SQL containing character length that is more than the SQL Server column default character length (6000).

dbt Cloud collector Rather than reporting an error, the collector now skips job runs that do not have generated documentation artifacts.

Tableau collector properly catalogs all sites if no site is specified in the CLI/YAML.

Monte Carlo collector: The collector now:

BigQuery collector:

Monte Carlo collector: Properly handles external URLs for tables that contain spaces.

dbt Core and dbt Cloud collectors: Properly harvests all meta config containing object values.

Profiling: Properly handles histogram values containing excessive range, overflow, or underflow values.

Go here to access release notes for previous version.