Release notes for previous versions

The following two new collectors are now available in public preview.

Power BI collector: The collector now harvests column descriptions for Power BI columns.

Databricks collector: The collector now supports Oauth service principal authentication for Databricks. Two new parameters, Service principal client ID (--client-id) and Service principal client secret (--client-secret) are introduced for this.

Tableau collector: Fixed an issue with the display of lineage between Tableau Fields and Database Columns, ensuring accurate representation of data relationships.

Snowflake Collector: The collector now harvests system tags.

SSIS collector: The collector now supports the inclusion or exclusion of specific databases or servers from being harvested. For new parameters are introduced to use these features: --include-database, --exclude-database, --include-server, --exclude-server.

All collectors: Resolved an issue where collectors created new collections with a new ID, leading to duplicate collections in the catalog.

Qlik Sense collector: Fixed an issue where missing user information in Qlik Sense resulted in an exception trace in the logfile.

Azure data factory collector: Added a log message to indicate when the Dataset API response lacks sufficient information, such as schema and table details, to construct lineage.

Tableau collector: The way we represent ownership information for workbooks, views, and metrics in the Tableau catalog has been updated. The owner is now represented using the kos:hasOwner property.

Alteryx collector: Increased API call read timeout and improved error handling by capturing and logging processing exceptions.

Confluent collectors: The collector now correctly handles cases where consumer member assignments are missing a topic description.

Tableau collector: The collector now accurately harvests hidden dashboards. Previously, the feature was limited to hidden Views, which included both Sheets and Dashboards but classified them all as Sheets. With this update, the collector distinguishes between Sheets and Dashboards, assigning the correct type to each entity. This ensures a more accurate representation of hidden Views in Tableau.

A new collector, the AWS Database Migration Service (DMS) collector, is now available in public preview.

Tableau collector: Improve command guidance, documentation, and warnings about the required format of the Tableau API URL option.

Alteryx collector: The collector now harvests nested workflow nodes and catalogs their relationship with the workflow.

Azure Data Factory collector: Made improvements to Azure Data Factory lineage by enhancing the harvesting of lineage from parameterized dataset references. The collector now also harvests both downstream and upstream resources.

Oracle collector: Add a new parameter --autonomous-db-connection-string for connection string for autonomous DB.

Tableau collector: The collector now harvests relationships between SQL tables and workbooks, enhancing data connectivity and visualization.

AWS Glue collector: Enhanced the collector to harvest lineage from Glue Data Catalog tables to their underlying S3 objects and gather more metadata for tables. The enhanced collector is available with the command catalog-aws-glue, while the legacy collector remains available as catalog-aws-glue-legacy or catalog-awsglue for compatibility. Please coordinate with your Customer Success Director for a smooth transition to the new collector version soon.

Note that the AWS Glue collector is only available as an on-premise solution, not as a cloud collector.

Tableau collector:

Tableau collector:

Databricks collector: Fixed an issue where a null pointer exception occurred while harvesting tags from Databricks.

Databricks collector: Fixed an exception that occurred when the table type did not match any known types.

Fivetran collector: Updated to use new APIs for retrieving column lineage due to changes in Fivetran API.

Snowflake collector: Added support for parsing SQL that utilizes IDENTIFIER() function calls when passing exact table names.

Tableau collector (Preview): Added a null check in custom SQL Table logic to prevent errors.

SQL Server collector: Reduced excessive log and warning messages during dependency collection to streamline output.

Databricks collector: Improved error handling for connection issues with the Databricks host.

Databricks collector: Added support for harvesting resources with the browse privilege.

Tableau collector: Now supports harvesting lineage between Custom SQL tables and their upstream tables.

Power BI and Power BI Gov collectors: Handling calculated tables as a new type and cataloging table-level lineage to source tables and columns.

Monte Carlo collector: Updated the collector to remove reaction type from incidents, as it has been deprecated in Monte Carlo GraphQL responses.

Salesforce collector: Added a null check to prevent exceptions when last modified by information is missing.

The following two new collectors are now available in public preview.

Reltio collector: Resolved an issue that caused errors when cataloging containment relationships for certain attribute containers

Power BI and Power BI Gov collectors: Stopped cataloging JDBC types for columns in Power BI, as this task is best handled by the database collector. Power BI previously attempted to infer JDBC types based on its column types, which is now corrected.

Power BI and Power BI Gov collectors:

Tableau collector: Enhanced the collector to catalog unpublished views, expanding visibility into Tableau assets.

Snowflake collector: Enhanced the incremental collection process to prevent the deletion of specific database column resources, ensuring data integrity and continuity.

Monte Carlo collector: Updated to catalog Monte Carlo warehouse IDs instead of hostnames due to API changes.

Databricks collector: Fixed an issue where the --include-information-schema option produced an incorrect warning.

Denodo collector: Resolved an issue with the internal cleanup function that was unable to remove PRIMARY KEY statements from SQL.

Power BI Service and Power BI Gov collectors: Resolved an issue with handling parameters when they are referenced using @ symbol.

Databricks collector: Added support for harvesting SQL queries and their associated lineage. A new parameter Page size for harvesting queries (--query-pagination-limit) is introduced for this.

Power BI Service and Power BI Gov collectors: Added support for jdbcProperties in the datasources.yaml configuration for connecting to databases when resolving lineage.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Database authentication issues are now reported as errors.

Oracle collector:

Redshift collector: The collector now correctly harvests distinct utility functions and procedures.

Databricks collector: Resolved an issue where column names were not recognized due to case sensitivity mismatches.

Oracle Collector: Now supports harvesting from Oracle Autonomous Database. A new parameter --autonomous-db is introduced for this.

Tableau collector: Now supports harvesting of personal space workbooks. A new parameter --tableau-catalog-personal-space-workbooks is introduced for this.

Power BI collector: Resolved an issue with parameter replacements in table source expressions when a parameter name is the same as the name of the table it defines.

SQL Server collector: Ensured encryption is enabled for connections when the encrypt JDBC property is configured.

Alteryx collector: Fixed an issue encountered while fetching workflow details when the user does not have permission on the workflow.

Power BI Service and Power BI gov collectors:

Snowflake, Redshift, Databricks, Oracle, PostgreSQL, Db2, Netezza, SQL Server collectors: Improved handling of lineage harvesting from SQL statements that contain quoted dashes, which were previously misinterpreted as comments.

Oracle collector: Corrected the handling of dependency harvesting to focus only on requested schemas.

Snowflake, Oracle, and SQL Server collectors: Modified to query dependencies across the entire database at once, rather than by individual schema, to reduce the number of queries executed.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Introduced a new parameter, --exclude-schema, to exclude specific schemes from the collection process.

Databricks collector: Added support for harvesting notebook content and lineage.

Snowflake collector: Enhanced support for harvesting lineage from view select statements that include QUALIFY and TRY_CAST constructs.

Redshift collector: The collector is now able to harvest definitions (DDL) for stored procedures and functions.

All Collectors: Improved the collectors logging for greater consistency in local and uploaded logfile names. Updated log file naming conventions to ensure uniqueness and preserve log files from previous runs when uploading to data.world.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Enhanced support for harvesting lineage from view select statements containing a mix of qualified and unqualified table names.

dbt Core and dbt Cloud collectors: The relationship between DbtSource and its abstracted tables or views is now defined as representsDataSource. This change better reflects the semantics of dbt sources, and improves visualization of lineage relationships in Eureka Explorer.

Tableau Collector: Released an updated collector for Tableau, featuring improved detection of lineage relationships to database objects along with stability and performance enhancements. While the new collector will co-exist with the legacy version, we encourage transitioning to the new version to take advantage of these enhancements.

Snowflake collector: Added support for additional SQL syntaxes when parsing statements that include the QUALIFY keyword.

Alteryx collector: Added support for honoring the max job limit option when set to zero.

Snowflake collector: Resolved an issue that caused duplicate schema processing and duplicate CatalogCuration resources during incremental collection.

Snowflake collector:

PostgreSQL collector: Added support for AWS IAM Authentication tokens for databases hosted on AWS.

Power BI Service and Power BI Gov collectors: Updated to accommodate Microsoft’s transition from dataset to semantic model in all catalog resources emitted by the collector.

Redshift collector: Corrected handling of lineage harvesting from SQL statements using ARRAY literals.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Improved handling of lineage harvesting from SQL statements containing KEYS as a column name.

Snowflake collector: Fixed issues in harvesting lineage from SQL statements using QUALIFY and COPY GRANTS keywords.

Power BI Service and Power BI Gov collectors: Introduced various improvements to the datasource template.

Denodo collector: Improved column fetching by processing one view or table at a time if an issue occurs when retrieving all columns at once.

Fivetran collector: Added support for Salesforce as a source.

Salesforce collector: Added support for harvesting metadata for summary (roll-up) fields.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Resolved an issue by clearing lineage resolution cache between runs to avoid conflicts when multiple commands are configured.

SQL Server collector: Added support for Active Directory Service Principal and Entra ID authentications.

Amazon S3 collector: Enhanced object filtering based on configuration options to include or exclude object names before checking the maximum resources limit.

AWS Glue Collector: Added functionality to catalog partitioned columns.

Power BI Service and Power BI Gov collectors: Resolved an issue with the CombineColumns transform step that was causing warnings in some cases due to misalignment in lineage to source columns.

dbt Core collector: Improved handling of situations where the profiles.yml file is mistakenly specified as a directory.

Salesforce collector: The security token configuration option is no longer required, as there are scenarios where authentication to the Salesforce API does not require it.

Power BI Service and power BI Gov collectors: Fixed a problem that was occurring while attempting to replace parameters in custom SQL when the parameter name contained a special character.

dbt cloud collector: Fixed an issue in which the user-specified Snowflake account override was being ignored by the collector.

Reltio collector: The collector now supports client_credentials authentication, in addition to the existing user and password authentication.

Denodo collector: The collector now harvests primary and foreign keys by schema to improve performance.

Databricks collector: Introduced a new parameter, --exclude-system-functions, which allows collector to exlude harvesting of built-in Databricks system functions.

Power BI Service and Power BI Gov collectors: The collectors now catalog the following additional properties:

Oracle collector: Modified the --linked-host option to require the database name for a linked database, as there was no reliable way to query this through the link.

Power BI Service and Power BI Gov collectors: Fixed an issue with parsing parameters in SQL to account for the use of Text.From().

The following two new collectors are now available in private preview. If you would like access to this collector, please contact your Customer Success Director.

Azure Data Lake Storage Gen2 collector: Enabled harvesting of Azure blob storage. Two new parameters are introduced as a result of this change: --disable-adls-gen2, --disable-azure-blob-storage

PowerBI collector:

ADLS (Azure Data Lake Storage) Gen2 collector: The collector now supports harvesting Azure blob storage.

Redshift collector: Added support for harvesting materialized views.

Databricks collector: Skipped collection of extended table metadata for system database to ensure stable run.

Monte Carlo collector: Resolved a null pointer error caused by unexpected null values.

PowerBI collector: The collector now supports parsing custom SQL queries for Databricks sources.

Oracle collector:

Databricks collector:

SQL Server Integration Services (SSIS) collector: Fixed an issue with missing file connection strings.

SQL Server collector: Added support for parsing WITH TIES and TRY_CAST statements.

PowerBI collector: Added an optional starter datasource YAML file required for harvesting lineage. If the YAML file is missing or incomplete, a template will be generated in the collector logs for easy reference.

All collectors: Fixed an issue with detecting the existence of configured datasets.

SQL Server collector now catalogs:

PowerBI collector: Improved data source titles for better specificity.

Salesforce collector: Optimized the process of fetching authentication tokens to reduce the number of Salesforce API calls.

Sigma collector: Fixed an issue with API structure changes due to pagination limits.

Redshift collector:  Resolved an issue where the collector incorrectly reported that the provided credential did not have access to the database tables being harvested.

Snowflake collector:

Denodo collector:

SQL Server collector: Added support for lineage in stored procedures that do not use BEGIN/END statements in the procedure body.

dbt Cloud collector: The collector now accommodates large numeric values for account, project, and run identifiers.

Redshift collector: Fixed an issue with view parsing for lineage when no schema binding is present.

The following two new collectors are now available in private preview. If you would like access to this collector, please contact your Customer Success Director.

The Amazon QuickSight collector is now generally available to all customers.

Power BI Report Server (PBIRS) collector: Fixed an issue when collector resource does not have a parent folder.

Oracle collector: The collector now supports cataloging lineage when database links are used in view and stored procedure and function definitions.

Snowflake collector: The collector now properly handles comments in view SQL definition during lineage parsing.

Sigma collector: Resolved an error caused by changes in the API response structure when the number of records exceed the default pagination limit.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: The collectors now support harvesting decimal digits metadata for columns.

Generic JDBC, Denodo, MySQL, SQL Server collectors: Resolved arithmetic overflow error when converting expressions to data type bigint.

SSIS collector: Added database location information when creating a database asset.

Tableau collector: The collector now supports filtering descendant projects.

Salesforce collector: The collector now properly handles null responses from the Salesforce API.

Snowflake collector: Fixed an issue where column nodes were not being copied to the interleaved Snowflake graph.

Tableau collector: Fixed an issue with missing owner users for various resources.

Power BI and Power BI Gov collectors: Added new relationships from app report to workspace report.

Salesforce Collector: Implemented a fix to avoid exceptions when custom object field metadata contains null values.

Denodo Collector: Resolved an issue with harvesting view SQL.

Power BI Service and Power BI Gov collectors:

Power BI Gov collector: The collector now supports harvesting of user workspaces using the --include-user-workspace parameter.

Databricks collector: The collector now supports harvesting data from ADLS (Azure Data Lake Storage) Gen2 external location.

dbt Core collector: Added support for dbt projects using the dbt-sqlserver adapter.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue that occurred while fetching information on data sources for a report.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue that caused API requests to return an HTTP 404 error.

SQL Server collector: If an error occurs while fetching columns from the database by schema, the collector now attempts to fetch columns by table instead.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue when collector resources were not returned by the API.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue where the collector would terminate abnormally if the SSRS API returned no data under certain circumstances.

Power BI Service and Power BI Gov Collectors: The collectors now correctly handled case mismatches in source column names when resolving SQL statements for lineage.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue that caused the collector to terminate unexpectedly when encountering Linked Reports with names containing non-alphanumeric characters.

Oracle collector: Enabled caching for primary keys and foreign keys, and reduced the number of queries used to gather table and column extended metadata, resulting in improved collector run time.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Item path is now harvested for report, data source, and dataset titles.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Resolved an issue with NTLM Authentication.

Power BI Service and Power BI Gov collectors: The collectors now support TNS connection strings in lineage parsing for Oracles sources if HOST and SID are specified. For example, (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=SERVER_NAME)(PORT=1521))(CONNECT_DATA=(SID=KOSTEST))).

SQL Server Reporting Services and Power BI Report Service collectors: The collectors now support authentication using NTLM.

Amazon S3 collector: The collector now harvests objects that begin with a prefix.

Salesforce collector: The collector now harvests metadata for Objects, Fields, Dashboards, and Reports. It also supports OAuth authentication instead of Basic authentication. You must complete the new pre-requisite tasks to use OAuth authentication.

Tableau collector: Enhanced resiliency for Tableau GraphQL query execution.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Power BI Service and Power BI Gov collectors: Fixed an issue with handling parameters that are defined in the tables section of Semantic Models, allowing for successful parsing of source information for tables using those parameters.

Snowflake collector: The collector now appropriately handles date time parsing for the timestamp NTZ format.

Power BI Service and Power BI Gov collectors: The collectors now support Denodo sources in Power BI column-level lineage parsing.

Denodo collector: The collector now harvests column-level lineage.

SQL Server Integration Services (SSIS) collector: Added a new --jdbc-property parameter. This allows you to provide authentication details for NTLM Authentication type.

dbt Core and dbt Cloud collectors: The collectors now harvests model columns from catalog.json and manifest.json database objects.

Power BI collector: The collector now properly handle scenarios where columns are renamed in Power BI that resulted in duplicate columns in source tables.

Azure Data Factory collector: The collector now properly performs Date transformation when the time zone is not available as ZoneID.

Azure Data Lake Storage Gen2 collector:

Power BI Gov collector: The collector now harvests preview images for Power BI reports. Add the new parameter --image-collection to your command/YAML file to use this new feature.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Snowflake collector: Resolved an issue where the collector was cataloging an incorrect database when the user had a default namespace set in Snowflake.

Databricks collector: Fixed an issue where the collector output files uploads were failing due to spaces in IRIs.

QuickSight Collector: Fixed an AwsAccountId null error while listing resources using pagination, which was causing issues in cataloging all the specified resources.

Azure Data Factory collector:

Oracle collector: Added support for lineage when the SELECT statement contains synonyms. This enhancement fixes lineage tracking between Oracle and Power BI when synonyms are used.

Power BI collector: The collector now harvests preview images for Power BI reports. Add the new parameter --image-collection to your command/YAML file to use this new feature.

Power BI and Power BI Gov collectors:

All collectors: Each catalog resource in the catalog output file now contains information about the collector that harvested the resource. This information is available only in the catalog file and can be used in SPARQL automations.

Power BI and Power BI Gov collectors: The collectors now properly handle scenarios when they run into API request limits. A new parameter Disable max requests wait (--disable-max-requests-wait) is added for handling these scenarios.

Azure Data Lake Storage Gen2 collector: Resolved an issue where certain ACL information missing in the Azure Data Lake Storage API response caused errors in the collector.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Harvesting of column-level lineage from views now supports view definitions containing unaliased subselects.

Power BI and Power BI Gov collectors: The collectors now support parsing data source expressions for Power BI tables where the source connection information is defined as a parameter. This means that if Power BI users specify data source connection information in a parameter and use that parameter in place of the source in the expression, the collectors will correctly parse and resolve the expression/lineage.

Oracle collector: The collector now harvests from DBA_ views if the credential used to execute the collector lacks permissions for information schema views.

dbt Core collector: The collector now harvest database objects and intra-database lineage from dbt projects and artifacts that use Azure Synapse as a backend.

All collectors: Collectors now verify that the user-requested upload location exists with proper permissions before execution and issue a warning if a problem exists.

Databricks collector: The collector no longer supports Databricks-managed password authentication. If you used this method of authentication, you must change the authentication to personal access token. For details, see "Preparing Databricks for collectors".

SQL Server collector:

Power BI and Power BI Gov collectors: Fixed an issue where logging operations were causing an exception if certain Power BI objects were null.

Tableau collector: Fixed an issue where certain Tableau projects were not fully cataloged.

Power BI Gov Collector:

Azure Data Factory collector: The collector now harvests Expressions for table names, schema names, file names.

A new collector for SQL Server Integration Services (SSIS) is now available in private preview. If you would like access to this collector, please contact your Customer Success Director.

Power BI and Power BI Gov collectors: The collectors now correctly harvest lineage for column types.

Snowflake collector: The collector now harvests metadata for functions and stored procedures from the snowflake.account_usage views when the metadata is unavailable from the information_schema of the database.

Power BI and Power BI gov collectors now catalog:

ADF collector: Fixed an issue with datetime parse errors while harvesting triggers.

Power BI and Power BI gov collectors: The collectors now support lineage for Oracle database objects.

Power BI and Power BI gov collectors: Resolved an issue with collecting child resources for apps when using service principal authentication.

Snowflake and Oracle collectors: Fixed an issue where the system now correctly does not harvest function lineage when users enable the Disable lineage collection (--disable-lineage-collection) option.

Oracle collector: Fixed an issue with harvesting database columns of LONG type.

Power BI and Power BI Gov collectors:

Databricks collector: The collector now harvests table and column tags by schema.

Snowflake collector was unable to harvest lineage if the SQL statement included a dash in the column aliases.

Snowflake, Teradata, Netezza collectors: Fixed an issue that occurred because of insufficient information while harvesting agent resources for functions and procedures.

SQL Server collector: Fixed an issue that occurred while parsing view queries where columns have dashes in their names.

Databricks collector: The collector now harvests table and column lineage from system tables. To use this feature, you need to set new permissions for the collector.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Resolved a problem concerning column statistics when an aggregate statistic has a zero value.

Tableau collector: Resolved an issue to correctly associate lineage with the appropriate parent project.

Sigma collector: Resolved an issue which occurred when a dataset referred to in the lineage was not available among the harvested datasets.

Snowflake collector: Fixed an issue associated with external URLs containing special characters.

Snowflake collector:

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

BigQuery collector: The collector is updated to generate catalog records for BigQuery Label instances. This allows them to be visible on the resource pages in the application.

Sigma collector: Resolved an issue that could result in an exception when the Sigma APIs failed to return a table path for a connection.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Sigma collector:

Snowflake collector: The collector now harvests extended metadata for tables, views, and materialized views.

SQL Server collector: Incorporated additional debug logging for when the collector fails to harvest extended metadata.

Oracle collector:

Power BI and Power BI Gov collectors: The OBDC data sources YAML file (datasources.yml) is updated to allow user-specified aliases for the database location (host). This ensures that resources are accurately linked across collectors.

Snowflake collector: Added support for harvesting materialized views for SQL definition, External URL (Snowsight).

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

SQL Server collector: The collector now correctly manages a scenario to use a consistent case when a collation is set.

dbt core and dbt cloud collectors: The collectors are optimized to correctly manage scenarios that previously caused an exception while harvesting lineage.

Sigma collector: The collector is optimized to manage scenarios that were previously causing the collector to not run properly.

dbt Core collector:

dbt cloud collector now comes with enhancements that optimize the harvesting of column-level lineage for dbt models.

Sigma collector properly deserializes objects from Sigma API.

Power BI and Power BI gov collectors now properly obtains server name and port from Power BI data source parameters.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Oracle Collector: Added a new --oracle-jdbc-timezone-as-region parameter. This allows you to decide if the Oracle JDBC connection timezone should utilize the JVM's default timezone.

AWS Glue Collector: Improved the log message that are recorded when the harvesting of job lineage fail.

Oracle and SQL Server collectors: The collectors now catalog column-level lineage when functions and stored procedures contain sub-selects.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Power BI and Power BI gov collectors now catalog:

Teradata collector properly harvests lineage metadata from views with SQL statements containing REPLACE RECURSIVE VIEW, LOCK ROW ACCESS.

Oracle collector properly harvests lineage metadata from views with COLLECT.

All collectors properly handle config file options that start with option flags.

All collectors: Users now have the option to define a custom output file name for the collector catalog during run time. To do this, use the --output-name parameter. The system automatically adds .dwec.ttl to the end of the provided file name.

Oracle Collector: The collector now harvests Oracle package bodies and Oracle package specifications.

SQL Server collector Fixed an error that occurred when harvesting column statistics.

Power BI and Power BI Gov collectors: Resolved an issue that was causing errors during the parsing of expressions that used the Table.RenameColumns Power Query table function in certain cases.

Snowflake Collector: The collector now properly harvest tags that are defined in a different schema than the schemas specified for the collector.

The following collectors are updated to harvest lineage accurately for group by, order by, where, and having SQL expressions. Prior to this update, the relationships were incorrectly directed.

Postgres, Databricks, Derby, Netezza, Oracle, Redshift, Snowflake, SQL Server, Teradata collectors

A new collector is now available for Amazon Managed Streaming for Kafka.

Oracle collector: The collector now harvests lineage from views, stored procedures, and functions.

Snowflake collector: The collector now harvests Streamlit apps.

The following collectors now support harvesting from multiple databases specified by users. This means you can provide the --database parameter multiple times while running the collector.

Power BI and Power BI Gov collector: Resolved an issue that was caused by parsing expand column expressions.

dbt cloud collector: The collector now properly harvests metadata of dbt Cloud artifacts when the target database is not Snowflake. Note the collector will only harvest metadata from the dbt Cloud artifacts and not connect to any unsupported target database to obtain database lineage metadata.

Snowflake collector: The collector harvest policies associated with cataloged database objects, regardless of the database in which the policies reside.

Oracle collector: The collector now harvest Synonyms.

Athena collector: Starting with release 2.198, data.world no longer package the Athena JDBC driver with the Athena collector. You can continue to use the releases previous to 2.198 as-is, but when you updated to the collector version to 2.198 or higher, you will have to download and mount the driver for the collector and update the collector command to include the driver path.

Log files for collectors: The collector log files for each collector run now have unique names. This allows logs to be written to separate files when running multiple collector instances.

Reltio collector: Survivorship groups and mappings are now recognized as primary entities with catalog records.

Snowflake collector: The collector now harvests tags associated with database objects in the user-specified database, regardless of the database in which the tag resides.

Teradata collector: Fixed an issue that was blocking column harvesting due to invalid column references in Views.

Azure data Factory collector: Fixed an issue preventing successful file uploads to data.world.

Databricks collector: The collector now harvests tags for Databases, Schemas, Tables, and Columns.

Power BI Service and Power BI Gov collectors: The collectors now correctly harvest skipped data sources during metadata scans.

Azure Data Lake Storage Gen2 collector: The collector is updated to refresh API authorization requests per ADLS requirements to avoid session expiration.

Azure Data Factory collector: Fixed an issue to accommodate varying data returned from the Azure Data Factory API.

The Power BI Service and Power BI Gov collectors now support harvesting lineage from ODBC data source types. A new parameter --datasource-mapping-file  can be used to provide the information required for harvesting lineage relationships when the data source uses an ODBC connection in Power BI.

The Amazon S3 collector now continues to harvest objects in the bucket when a 403 error is encountered.

The Azure Data Lake Storage Gen2 collector properly handles the scenario involving special characters in the blob name.

The Azure Data Factory collector properly handles a scenario that causes the collector to stop due to the format of information returned from the Azure Data Factory APIs.

BigQuery Collector properly handles a scenario when a table is in a different database from the one being harvested.