Catalog collector release notes

Databricks collector: Added support for harvesting SQL queries and their associated lineage. A new parameter Page size for harvesting queries (--query-pagination-limit) is introduced for this.

Power BI Service and Power BI Gov collectors: Added support for jdbcProperties in the datasources.yaml configuration for connecting to databases when resolving lineage.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Database authentication issues are now reported as errors.

Oracle collector:

Redshift collector: The collector now correctly harvests distinct utility functions and procedures.

Databricks collector: Resolved an issue where column names were not recognized due to case sensitivity mismatches.

Oracle Collector: Now supports harvesting from Oracle Autonomous Database. A new parameter --autonomous-db is introduced for this.

Tableau collector: Now supports harvesting of personal space workbooks. A new parameter --tableau-catalog-personal-space-workbooks is introduced for this.

Power BI collector: Resolved an issue with parameter replacements in table source expressions when a parameter name is the same as the name of the table it defines.

SQL Server collector: Ensured encryption is enabled for connections when the encrypt JDBC property is configured.

Alteryx collector: Fixed an issue encountered while fetching workflow details when the user does not have permission on the workflow.

Power BI Service and Power BI gov collectors:

Snowflake, Redshift, Databricks, Oracle, PostgreSQL, Db2, Netezza, SQL Server collectors: Improved handling of lineage harvesting from SQL statements that contain quoted dashes, which were previously misinterpreted as comments.

Oracle collector: Corrected the handling of dependency harvesting to focus only on requested schemas.

Snowflake, Oracle, and SQL Server collectors: Modified to query dependencies across the entire database at once, rather than by individual schema, to reduce the number of queries executed.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Introduced a new parameter, --exclude-schema, to exclude specific schemes from the collection process.

Databricks collector: Added support for harvesting notebook content and lineage.

Snowflake collector: Enhanced support for harvesting lineage from view select statements that include QUALIFY and TRY_CAST constructs.

Redshift collector: The collector is now able to harvest definitions (DDL) for stored procedures and functions.

All Collectors: Improved the collectors logging for greater consistency in local and uploaded logfile names. Updated log file naming conventions to ensure uniqueness and preserve log files from previous runs when uploading to data.world.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Enhanced support for harvesting lineage from view select statements containing a mix of qualified and unqualified table names.

dbt Core and dbt Cloud collectors: The relationship between DbtSource and its abstracted tables or views is now defined as representsDataSource. This change better reflects the semantics of dbt sources, and improves visualization of lineage relationships in Eureka Explorer.

Tableau Collector: Released an updated collector for Tableau, featuring improved detection of lineage relationships to database objects along with stability and performance enhancements. While the new collector will co-exist with the legacy version, we encourage transitioning to the new version to take advantage of these enhancements.

Snowflake collector: Added support for additional SQL syntaxes when parsing statements that include the QUALIFY keyword.

Alteryx collector: Added support for honoring the max job limit option when set to zero.

Snowflake collector: Resolved an issue that caused duplicate schema processing and duplicate CatalogCuration resources during incremental collection.

Snowflake collector:

PostgreSQL collector: Added support for AWS IAM Authentication tokens for databases hosted on AWS.

Power BI Service and Power BI Gov collectors: Updated to accommodate Microsoft’s transition from dataset to semantic model in all catalog resources emitted by the collector.

Redshift collector: Corrected handling of lineage harvesting from SQL statements using ARRAY literals.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Improved handling of lineage harvesting from SQL statements containing KEYS as a column name.

Snowflake collector: Fixed issues in harvesting lineage from SQL statements using QUALIFY and COPY GRANTS keywords.

Power BI Service and Power BI Gov collectors: Introduced various improvements to the datasource template.

Denodo collector: Improved column fetching by processing one view or table at a time if an issue occurs when retrieving all columns at once.

Fivetran collector: Added support for Salesforce as a source.

Salesforce collector: Added support for harvesting metadata for summary (roll-up) fields.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Resolved an issue by clearing lineage resolution cache between runs to avoid conflicts when multiple commands are configured.

SQL Server collector: Added support for Active Directory Service Principal and Entra ID authentications.

Amazon S3 collector: Enhanced object filtering based on configuration options to include or exclude object names before checking the maximum resources limit.

AWS Glue Collector: Added functionality to catalog partitioned columns.

Power BI Service and Power BI Gov collectors: Resolved an issue with the CombineColumns transform step that was causing warnings in some cases due to misalignment in lineage to source columns.

dbt Core collector: Improved handling of situations where the profiles.yml file is mistakenly specified as a directory.

Salesforce collector: The security token configuration option is no longer required, as there are scenarios where authentication to the Salesforce API does not require it.

Power BI Service and power BI Gov collectors: Fixed a problem that was occurring while attempting to replace parameters in custom SQL when the parameter name contained a special character.

dbt cloud collector: Fixed an issue in which the user-specified Snowflake account override was being ignored by the collector.

Reltio collector: The collector now supports client_credentials authentication, in addition to the existing user and password authentication.

Denodo collector: The collector now harvests primary and foreign keys by schema to improve performance.

Databricks collector: Introduced a new parameter, --exclude-system-functions, which allows collector to exlude harvesting of built-in Databricks system functions.

Power BI Service and Power BI Gov collectors: The collectors now catalog the following additional properties:

Oracle collector: Modified the --linked-host option to require the database name for a linked database, as there was no reliable way to query this through the link.

Power BI Service and Power BI Gov collectors: Fixed an issue with parsing parameters in SQL to account for the use of Text.From().

The following two new collectors are now available in private preview. If you would like access to this collector, please contact your Customer Success Director.

Azure Data Lake Storage Gen2 collector: Enabled harvesting of Azure blob storage. Two new parameters are introduced as a result of this change: --disable-adls-gen2, --disable-azure-blob-storage

PowerBI collector:

ADLS (Azure Data Lake Storage) Gen2 collector: The collector now supports harvesting Azure blob storage.

Redshift collector: Added support for harvesting materialized views.

Databricks collector: Skipped collection of extended table metadata for system database to ensure stable run.

Monte Carlo collector: Resolved a null pointer error caused by unexpected null values.

PowerBI collector: The collector now supports parsing custom SQL queries for Databricks sources.

Oracle collector:

Databricks collector:

SQL Server Integration Services (SSIS) collector: Fixed an issue with missing file connection strings.

SQL Server collector: Added support for parsing WITH TIES and TRY_CAST statements.

PowerBI collector: Added an optional starter datasource YAML file required for harvesting lineage. If the YAML file is missing or incomplete, a template will be generated in the collector logs for easy reference.

All collectors: Fixed an issue with detecting the existence of configured datasets.

SQL Server collector now catalogs:

PowerBI collector: Improved data source titles for better specificity.

Salesforce collector: Optimized the process of fetching authentication tokens to reduce the number of Salesforce API calls.

Sigma collector: Fixed an issue with API structure changes due to pagination limits.

Redshift collector:  Resolved an issue where the collector incorrectly reported that the provided credential did not have access to the database tables being harvested.

Go here to access release notes for previous version.