Catalog collector release notes

Snowflake Collector: The collector now harvests system tags.

SSIS collector: The collector now supports the inclusion or exclusion of specific databases or servers from being harvested. For new parameters are introduced to use these features: --include-database, --exclude-database, --include-server, --exclude-server.

All collectors: Resolved an issue where collectors created new collections with a new ID, leading to duplicate collections in the catalog.

Qlik Sense collector: Fixed an issue where missing user information in Qlik Sense resulted in an exception trace in the logfile.

Azure data factory collector: Added a log message to indicate when the Dataset API response lacks sufficient information, such as schema and table details, to construct lineage.

Tableau collector: The way we represent ownership information for workbooks, views, and metrics in the Tableau catalog has been updated. The owner is now represented using the kos:hasOwner property.

Alteryx collector: Increased API call read timeout and improved error handling by capturing and logging processing exceptions.

Confluent collectors: The collector now correctly handles cases where consumer member assignments are missing a topic description.

Tableau collector: The collector now accurately harvests hidden dashboards. Previously, the feature was limited to hidden Views, which included both Sheets and Dashboards but classified them all as Sheets. With this update, the collector distinguishes between Sheets and Dashboards, assigning the correct type to each entity. This ensures a more accurate representation of hidden Views in Tableau.

A new collector, the AWS Database Migration Service (DMS) collector, is now available in public preview.

Tableau collector: Improve command guidance, documentation, and warnings about the required format of the Tableau API URL option.

Alteryx collector: The collector now harvests nested workflow nodes and catalogs their relationship with the workflow.

Azure Data Factory collector: Made improvements to Azure Data Factory lineage by enhancing the harvesting of lineage from parameterized dataset references. The collector now also harvests both downstream and upstream resources.

Oracle collector: Add a new parameter --autonomous-db-connection-string for connection string for autonomous DB.

Tableau collector: The collector now harvests relationships between SQL tables and workbooks, enhancing data connectivity and visualization.

AWS Glue collector: Enhanced the collector to harvest lineage from Glue Data Catalog tables to their underlying S3 objects and gather more metadata for tables. The enhanced collector is available with the command catalog-aws-glue, while the legacy collector remains available as catalog-aws-glue-legacy or catalog-awsglue for compatibility. Please coordinate with your Customer Success Director for a smooth transition to the new collector version soon.

Note that the AWS Glue collector is only available as an on-premise solution, not as a cloud collector.

Tableau collector:

Tableau collector:

Databricks collector: Fixed an issue where a null pointer exception occurred while harvesting tags from Databricks.

Databricks collector: Fixed an exception that occurred when the table type did not match any known types.

Fivetran collector: Updated to use new APIs for retrieving column lineage due to changes in Fivetran API.

Snowflake collector: Added support for parsing SQL that utilizes IDENTIFIER() function calls when passing exact table names.

Tableau collector (Preview): Added a null check in custom SQL Table logic to prevent errors.

SQL Server collector: Reduced excessive log and warning messages during dependency collection to streamline output.

Databricks collector: Improved error handling for connection issues with the Databricks host.

Databricks collector: Added support for harvesting resources with the browse privilege.

Tableau collector: Now supports harvesting lineage between Custom SQL tables and their upstream tables.

Power BI and Power BI Gov collectors: Handling calculated tables as a new type and cataloging table-level lineage to source tables and columns.

Monte Carlo collector: Updated the collector to remove reaction type from incidents, as it has been deprecated in Monte Carlo GraphQL responses.

Salesforce collector: Added a null check to prevent exceptions when last modified by information is missing.

The following two new collectors are now available in public preview.

Reltio collector: Resolved an issue that caused errors when cataloging containment relationships for certain attribute containers

Power BI and Power BI Gov collectors: Stopped cataloging JDBC types for columns in Power BI, as this task is best handled by the database collector. Power BI previously attempted to infer JDBC types based on its column types, which is now corrected.

Power BI and Power BI Gov collectors:

Tableau collector: Enhanced the collector to catalog unpublished views, expanding visibility into Tableau assets.

Snowflake collector: Enhanced the incremental collection process to prevent the deletion of specific database column resources, ensuring data integrity and continuity.

Monte Carlo collector: Updated to catalog Monte Carlo warehouse IDs instead of hostnames due to API changes.

Databricks collector: Fixed an issue where the --include-information-schema option produced an incorrect warning.

Denodo collector: Resolved an issue with the internal cleanup function that was unable to remove PRIMARY KEY statements from SQL.

Power BI Service and Power BI Gov collectors: Resolved an issue with handling parameters when they are referenced using @ symbol.

Databricks collector: Added support for harvesting SQL queries and their associated lineage. A new parameter Page size for harvesting queries (--query-pagination-limit) is introduced for this.

Power BI Service and Power BI Gov collectors: Added support for jdbcProperties in the datasources.yaml configuration for connecting to databases when resolving lineage.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Database authentication issues are now reported as errors.

Oracle collector:

Redshift collector: The collector now correctly harvests distinct utility functions and procedures.

Databricks collector: Resolved an issue where column names were not recognized due to case sensitivity mismatches.

Oracle Collector: Now supports harvesting from Oracle Autonomous Database. A new parameter --autonomous-db is introduced for this.

Tableau collector: Now supports harvesting of personal space workbooks. A new parameter --tableau-catalog-personal-space-workbooks is introduced for this.

Power BI collector: Resolved an issue with parameter replacements in table source expressions when a parameter name is the same as the name of the table it defines.

SQL Server collector: Ensured encryption is enabled for connections when the encrypt JDBC property is configured.

Alteryx collector: Fixed an issue encountered while fetching workflow details when the user does not have permission on the workflow.

Power BI Service and Power BI gov collectors:

Snowflake, Redshift, Databricks, Oracle, PostgreSQL, Db2, Netezza, SQL Server collectors: Improved handling of lineage harvesting from SQL statements that contain quoted dashes, which were previously misinterpreted as comments.

Oracle collector: Corrected the handling of dependency harvesting to focus only on requested schemas.

Snowflake, Oracle, and SQL Server collectors: Modified to query dependencies across the entire database at once, rather than by individual schema, to reduce the number of queries executed.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Introduced a new parameter, --exclude-schema, to exclude specific schemes from the collection process.

Databricks collector: Added support for harvesting notebook content and lineage.

Snowflake collector: Enhanced support for harvesting lineage from view select statements that include QUALIFY and TRY_CAST constructs.

Redshift collector: The collector is now able to harvest definitions (DDL) for stored procedures and functions.

All Collectors: Improved the collectors logging for greater consistency in local and uploaded logfile names. Updated log file naming conventions to ensure uniqueness and preserve log files from previous runs when uploading to data.world.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Enhanced support for harvesting lineage from view select statements containing a mix of qualified and unqualified table names.

dbt Core and dbt Cloud collectors: The relationship between DbtSource and its abstracted tables or views is now defined as representsDataSource. This change better reflects the semantics of dbt sources, and improves visualization of lineage relationships in Eureka Explorer.

Tableau Collector: Released an updated collector for Tableau, featuring improved detection of lineage relationships to database objects along with stability and performance enhancements. While the new collector will co-exist with the legacy version, we encourage transitioning to the new version to take advantage of these enhancements.

Snowflake collector: Added support for additional SQL syntaxes when parsing statements that include the QUALIFY keyword.

Alteryx collector: Added support for honoring the max job limit option when set to zero.

Snowflake collector: Resolved an issue that caused duplicate schema processing and duplicate CatalogCuration resources during incremental collection.

Go here to access release notes for previous version.