Catalog collector release notes

Microsoft Fabric collector:

Microsoft Fabric collector:

dbt Core and dbt Cloud collectors: dbt sources are now associated with database schemas (instead of tables) through a non-lineage relationship, improving metadata accuracy.

Tableau collector: Enhanced log tracing around column field lineage to improve troubleshooting and debugging clarity.

Databricks collector: Now captures table lineage to corresponding AWS S3 objects.

Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, SQL Server collectors: Added support for harvesting metadata about table indexes, providing deeper insight into database structure and performance optimization.

Tableau collector: Improved handling of missing connection types on Tableau Server to prevent ingestion errors and ensure smoother metadata extraction.

dbt collector: Transitioned from using the manifest to the catalog to retrieve database identifiers. This resolves issues with quoted identifiers (for example, Snowflake).

Athena, Amazon Database Migration Service (DMS), Amazon DynamoDB, AWS Glue, Amazon QuickSight, Amazon S3 collectors: The collectors now support multiple authentication methods, offering greater flexibility and compatibility. The following new parameters are introduced for the new authentication methods:

Amazon Database Migration Service (DMS) collector: Introduced support for S3 as both a source and target endpoint.

Tableau collector: Implemented a nonNull filter for relatedFields, addressing errors and improving stability in the Tableau collector.

Databricks collector: Fixed an issue where table lineage could not be resolved when information was incomplete.

Tableau collector: Corrected processing of column fields to resolve subtle errors in field-to-database lineage.

Azure data factory collector: Improved handling of missing table details when collecting lineage.

All collectors: Fixed logging issues where some messages were not being written to the log file as expected.

Monte Carlo collector: Fixed alignment issues in generated table information from Monte Carlo Monitor by properly escaping pipes ('|') in monitor names.

A new collector, the SAP HANA collector, is now available in public preview.

SSIS collector: Now captures the connections between packages through the control flow, improving visibility of package relationships.

Databricks collector: The collector now harvests lineage only within the current schema. A new option Harvest entire lineage (--harvest-entire-lineage) is added to enable harvesting lineage from external schemas.

Microsoft Fabric collector: Enhanced to support additional syntaxes for Semantic Model table connections to Fabric warehouse/lakehouse resources, broadening compatibility and connectivity.

QlikSense collector: Made performance improvements to increase the efficiency and speed of the collector.

Microsoft Fabric collector: Fixed issues with Lakehouses and SQL endpoints to ensure database resources are associated with the correct resource, enhancing accuracy.

Tableau collector: Fixed an issue to ensure proper handling of sites and projects with large quantities of column and data source fields.

Databricks collector: Resolved a parsing issue in view queries where aliases starting with a number caused failures.

The following two new collectors are now available in public preview.

Power BI collector: The collector now harvests column descriptions for Power BI columns.

Databricks collector: The collector now supports Oauth service principal authentication for Databricks. Two new parameters, Service principal client ID (--client-id) and Service principal client secret (--client-secret) are introduced for this.

Tableau collector: Fixed an issue with the display of lineage between Tableau Fields and Database Columns, ensuring accurate representation of data relationships.

Snowflake Collector: The collector now harvests system tags.

SSIS collector: The collector now supports the inclusion or exclusion of specific databases or servers from being harvested. For new parameters are introduced to use these features: --include-database, --exclude-database, --include-server, --exclude-server.

All collectors: Resolved an issue where collectors created new collections with a new ID, leading to duplicate collections in the catalog.

Qlik Sense collector: Fixed an issue where missing user information in Qlik Sense resulted in an exception trace in the logfile.

Azure data factory collector: Added a log message to indicate when the Dataset API response lacks sufficient information, such as schema and table details, to construct lineage.

Tableau collector: The way we represent ownership information for workbooks, views, and metrics in the Tableau catalog has been updated. The owner is now represented using the kos:hasOwner property.

Alteryx collector: Increased API call read timeout and improved error handling by capturing and logging processing exceptions.

Confluent collectors: The collector now correctly handles cases where consumer member assignments are missing a topic description.

Tableau collector: The collector now accurately harvests hidden dashboards. Previously, the feature was limited to hidden Views, which included both Sheets and Dashboards but classified them all as Sheets. With this update, the collector distinguishes between Sheets and Dashboards, assigning the correct type to each entity. This ensures a more accurate representation of hidden Views in Tableau.

A new collector, the AWS Database Migration Service (DMS) collector, is now available in public preview.

Tableau collector: Improve command guidance, documentation, and warnings about the required format of the Tableau API URL option.

Alteryx collector: The collector now harvests nested workflow nodes and catalogs their relationship with the workflow.

Azure Data Factory collector: Made improvements to Azure Data Factory lineage by enhancing the harvesting of lineage from parameterized dataset references. The collector now also harvests both downstream and upstream resources.

Oracle collector: Add a new parameter --autonomous-db-connection-string for connection string for autonomous DB.

Tableau collector: The collector now harvests relationships between SQL tables and workbooks, enhancing data connectivity and visualization.

AWS Glue collector: Enhanced the collector to harvest lineage from Glue Data Catalog tables to their underlying S3 objects and gather more metadata for tables. The enhanced collector is available with the command catalog-aws-glue, while the legacy collector remains available as catalog-aws-glue-legacy or catalog-awsglue for compatibility. Please coordinate with your Customer Success Director for a smooth transition to the new collector version soon.

Note that the AWS Glue collector is only available as an on-premise solution, not as a cloud collector.

Tableau collector:

Tableau collector:

Databricks collector: Fixed an issue where a null pointer exception occurred while harvesting tags from Databricks.

Databricks collector: Fixed an exception that occurred when the table type did not match any known types.

Fivetran collector: Updated to use new APIs for retrieving column lineage due to changes in Fivetran API.

Snowflake collector: Added support for parsing SQL that utilizes IDENTIFIER() function calls when passing exact table names.

Tableau collector (Preview): Added a null check in custom SQL Table logic to prevent errors.

SQL Server collector: Reduced excessive log and warning messages during dependency collection to streamline output.

Databricks collector: Improved error handling for connection issues with the Databricks host.

Databricks collector: Added support for harvesting resources with the browse privilege.

Tableau collector: Now supports harvesting lineage between Custom SQL tables and their upstream tables.

Power BI and Power BI Gov collectors: Handling calculated tables as a new type and cataloging table-level lineage to source tables and columns.

Monte Carlo collector: Updated the collector to remove reaction type from incidents, as it has been deprecated in Monte Carlo GraphQL responses.

Salesforce collector: Added a null check to prevent exceptions when last modified by information is missing.

Go here to access release notes for previous version.