Release notes for previous versions

Tableau collector: The collector now supports filtering descendant projects.

Salesforce collector: The collector now properly handles null responses from the Salesforce API.

Snowflake collector: Fixed an issue where column nodes were not being copied to the interleaved Snowflake graph.

Tableau collector: Fixed an issue with missing owner users for various resources.

Power BI and Power BI Gov collectors: Added new relationships from app report to workspace report.

Salesforce Collector: Implemented a fix to avoid exceptions when custom object field metadata contains null values.

Denodo Collector: Resolved an issue with harvesting view SQL.

Power BI Service and Power BI Gov collectors:

Power BI Gov collector: The collector now supports harvesting of user workspaces using the --include-user-workspace parameter.

Databricks collector: The collector now supports harvesting data from ADLS (Azure Data Lake Storage) Gen2 external location.

dbt Core collector: Added support for dbt projects using the dbt-sqlserver adapter.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue that occurred while fetching information on data sources for a report.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue that caused API requests to return an HTTP 404 error.

SQL Server collector: If an error occurs while fetching columns from the database by schema, the collector now attempts to fetch columns by table instead.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue when collector resources were not returned by the API.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue where the collector would terminate abnormally if the SSRS API returned no data under certain circumstances.

Power BI Service and Power BI Gov Collectors: The collectors now correctly handled case mismatches in source column names when resolving SQL statements for lineage.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Fixed an issue that caused the collector to terminate unexpectedly when encountering Linked Reports with names containing non-alphanumeric characters.

Oracle collector: Enabled caching for primary keys and foreign keys, and reduced the number of queries used to gather table and column extended metadata, resulting in improved collector run time.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Item path is now harvested for report, data source, and dataset titles.

SQL Server Reporting Services (SSRS) and Power BI Report Server (PBIRS) collectors: Resolved an issue with NTLM Authentication.

Power BI Service and Power BI Gov collectors: The collectors now support TNS connection strings in lineage parsing for Oracles sources if HOST and SID are specified. For example, (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=SERVER_NAME)(PORT=1521))(CONNECT_DATA=(SID=KOSTEST))).

SQL Server Reporting Services and Power BI Report Service collectors: The collectors now support authentication using NTLM.

Amazon S3 collector: The collector now harvests objects that begin with a prefix.

Salesforce collector: The collector now harvests metadata for Objects, Fields, Dashboards, and Reports. It also supports OAuth authentication instead of Basic authentication. You must complete the new pre-requisite tasks to use OAuth authentication.

Tableau collector: Enhanced resiliency for Tableau GraphQL query execution.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Power BI Service and Power BI Gov collectors: Fixed an issue with handling parameters that are defined in the tables section of Semantic Models, allowing for successful parsing of source information for tables using those parameters.

Snowflake collector: The collector now appropriately handles date time parsing for the timestamp NTZ format.

Power BI Service and Power BI Gov collectors: The collectors now support Denodo sources in Power BI column-level lineage parsing.

Denodo collector: The collector now harvests column-level lineage.

SQL Server Integration Services (SSIS) collector: Added a new --jdbc-property parameter. This allows you to provide authentication details for NTLM Authentication type.

dbt Core and dbt Cloud collectors: The collectors now harvests model columns from catalog.json and manifest.json database objects.

Power BI collector: The collector now properly handle scenarios where columns are renamed in Power BI that resulted in duplicate columns in source tables.

Azure Data Factory collector: The collector now properly performs Date transformation when the time zone is not available as ZoneID.

Azure Data Lake Storage Gen2 collector:

Power BI Gov collector: The collector now harvests preview images for Power BI reports. Add the new parameter --image-collection to your command/YAML file to use this new feature.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Snowflake collector: Resolved an issue where the collector was cataloging an incorrect database when the user had a default namespace set in Snowflake.

Databricks collector: Fixed an issue where the collector output files uploads were failing due to spaces in IRIs.

QuickSight Collector: Fixed an AwsAccountId null error while listing resources using pagination, which was causing issues in cataloging all the specified resources.

Azure Data Factory collector:

Oracle collector: Added support for lineage when the SELECT statement contains synonyms. This enhancement fixes lineage tracking between Oracle and Power BI when synonyms are used.

Power BI collector: The collector now harvests preview images for Power BI reports. Add the new parameter --image-collection to your command/YAML file to use this new feature.

Power BI and Power BI Gov collectors:

All collectors: Each catalog resource in the catalog output file now contains information about the collector that harvested the resource. This information is available only in the catalog file and can be used in SPARQL automations.

Power BI and Power BI Gov collectors: The collectors now properly handle scenarios when they run into API request limits. A new parameter Disable max requests wait (--disable-max-requests-wait) is added for handling these scenarios.

Azure Data Lake Storage Gen2 collector: Resolved an issue where certain ACL information missing in the Azure Data Lake Storage API response caused errors in the collector.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Harvesting of column-level lineage from views now supports view definitions containing unaliased subselects.

Power BI and Power BI Gov collectors: The collectors now support parsing data source expressions for Power BI tables where the source connection information is defined as a parameter. This means that if Power BI users specify data source connection information in a parameter and use that parameter in place of the source in the expression, the collectors will correctly parse and resolve the expression/lineage.

Oracle collector: The collector now harvests from DBA_ views if the credential used to execute the collector lacks permissions for information schema views.

dbt Core collector: The collector now harvest database objects and intra-database lineage from dbt projects and artifacts that use Azure Synapse as a backend.

All collectors: Collectors now verify that the user-requested upload location exists with proper permissions before execution and issue a warning if a problem exists.

Databricks collector: The collector no longer supports Databricks-managed password authentication. If you used this method of authentication, you must change the authentication to personal access token. For details, see "Preparing Databricks for collectors".

SQL Server collector:

Power BI and Power BI Gov collectors: Fixed an issue where logging operations were causing an exception if certain Power BI objects were null.

Tableau collector: Fixed an issue where certain Tableau projects were not fully cataloged.

Power BI Gov Collector:

Azure Data Factory collector: The collector now harvests Expressions for table names, schema names, file names.

A new collector for SQL Server Integration Services (SSIS) is now available in private preview. If you would like access to this collector, please contact your Customer Success Director.

Power BI and Power BI Gov collectors: The collectors now correctly harvest lineage for column types.

Snowflake collector: The collector now harvests metadata for functions and stored procedures from the snowflake.account_usage views when the metadata is unavailable from the information_schema of the database.

Power BI and Power BI gov collectors now catalog:

ADF collector: Fixed an issue with datetime parse errors while harvesting triggers.

Power BI and Power BI gov collectors: The collectors now support lineage for Oracle database objects.

Power BI and Power BI gov collectors: Resolved an issue with collecting child resources for apps when using service principal authentication.

Snowflake and Oracle collectors: Fixed an issue where the system now correctly does not harvest function lineage when users enable the Disable lineage collection (--disable-lineage-collection) option.

Oracle collector: Fixed an issue with harvesting database columns of LONG type.

Power BI and Power BI Gov collectors:

Databricks collector: The collector now harvests table and column tags by schema.

Snowflake collector was unable to harvest lineage if the SQL statement included a dash in the column aliases.

Snowflake, Teradata, Netezza collectors: Fixed an issue that occurred because of insufficient information while harvesting agent resources for functions and procedures.

SQL Server collector: Fixed an issue that occurred while parsing view queries where columns have dashes in their names.

Databricks collector: The collector now harvests table and column lineage from system tables. To use this feature, you need to set new permissions for the collector.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Resolved a problem concerning column statistics when an aggregate statistic has a zero value.

Tableau collector: Resolved an issue to correctly associate lineage with the appropriate parent project.

Sigma collector: Resolved an issue which occurred when a dataset referred to in the lineage was not available among the harvested datasets.

Snowflake collector: Fixed an issue associated with external URLs containing special characters.

Snowflake collector:

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

BigQuery collector: The collector is updated to generate catalog records for BigQuery Label instances. This allows them to be visible on the resource pages in the application.

Sigma collector: Resolved an issue that could result in an exception when the Sigma APIs failed to return a table path for a connection.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Sigma collector:

Snowflake collector: The collector now harvests extended metadata for tables, views, and materialized views.

SQL Server collector: Incorporated additional debug logging for when the collector fails to harvest extended metadata.

Oracle collector:

Power BI and Power BI Gov collectors: The OBDC data sources YAML file (datasources.yml) is updated to allow user-specified aliases for the database location (host). This ensures that resources are accurately linked across collectors.

Snowflake collector: Added support for harvesting materialized views for SQL definition, External URL (Snowsight).

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

SQL Server collector: The collector now correctly manages a scenario to use a consistent case when a collation is set.

dbt core and dbt cloud collectors: The collectors are optimized to correctly manage scenarios that previously caused an exception while harvesting lineage.

Sigma collector: The collector is optimized to manage scenarios that were previously causing the collector to not run properly.

dbt Core collector:

dbt cloud collector now comes with enhancements that optimize the harvesting of column-level lineage for dbt models.

Sigma collector properly deserializes objects from Sigma API.

Power BI and Power BI gov collectors now properly obtains server name and port from Power BI data source parameters.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Oracle Collector: Added a new --oracle-jdbc-timezone-as-region parameter. This allows you to decide if the Oracle JDBC connection timezone should utilize the JVM's default timezone.

AWS Glue Collector: Improved the log message that are recorded when the harvesting of job lineage fail.

Oracle and SQL Server collectors: The collectors now catalog column-level lineage when functions and stored procedures contain sub-selects.

Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors:

Power BI and Power BI gov collectors now catalog:

Teradata collector properly harvests lineage metadata from views with SQL statements containing REPLACE RECURSIVE VIEW, LOCK ROW ACCESS.

Oracle collector properly harvests lineage metadata from views with COLLECT.

All collectors properly handle config file options that start with option flags.

All collectors: Users now have the option to define a custom output file name for the collector catalog during run time. To do this, use the --output-name parameter. The system automatically adds .dwec.ttl to the end of the provided file name.

Oracle Collector: The collector now harvests Oracle package bodies and Oracle package specifications.

SQL Server collector Fixed an error that occurred when harvesting column statistics.

Power BI and Power BI Gov collectors: Resolved an issue that was causing errors during the parsing of expressions that used the Table.RenameColumns Power Query table function in certain cases.

Snowflake Collector: The collector now properly harvest tags that are defined in a different schema than the schemas specified for the collector.

The following collectors are updated to harvest lineage accurately for group by, order by, where, and having SQL expressions. Prior to this update, the relationships were incorrectly directed.

Postgres, Databricks, Derby, Netezza, Oracle, Redshift, Snowflake, SQL Server, Teradata collectors

A new collector is now available for Amazon Managed Streaming for Kafka.

Oracle collector: The collector now harvests lineage from views, stored procedures, and functions.

Snowflake collector: The collector now harvests Streamlit apps.

The following collectors now support harvesting from multiple databases specified by users. This means you can provide the --database parameter multiple times while running the collector.

Power BI and Power BI Gov collector: Resolved an issue that was caused by parsing expand column expressions.

dbt cloud collector: The collector now properly harvests metadata of dbt Cloud artifacts when the target database is not Snowflake. Note the collector will only harvest metadata from the dbt Cloud artifacts and not connect to any unsupported target database to obtain database lineage metadata.

Snowflake collector: The collector harvest policies associated with cataloged database objects, regardless of the database in which the policies reside.

Oracle collector: The collector now harvest Synonyms.

Athena collector: Starting with release 2.198, data.world no longer package the Athena JDBC driver with the Athena collector. You can continue to use the releases previous to 2.198 as-is, but when you updated to the collector version to 2.198 or higher, you will have to download and mount the driver for the collector and update the collector command to include the driver path.

Log files for collectors: The collector log files for each collector run now have unique names. This allows logs to be written to separate files when running multiple collector instances.

Reltio collector: Survivorship groups and mappings are now recognized as primary entities with catalog records.

Snowflake collector: The collector now harvests tags associated with database objects in the user-specified database, regardless of the database in which the tag resides.

Teradata collector: Fixed an issue that was blocking column harvesting due to invalid column references in Views.

Azure data Factory collector: Fixed an issue preventing successful file uploads to data.world.

Databricks collector: The collector now harvests tags for Databases, Schemas, Tables, and Columns.

Power BI Service and Power BI Gov collectors: The collectors now correctly harvest skipped data sources during metadata scans.

Azure Data Lake Storage Gen2 collector: The collector is updated to refresh API authorization requests per ADLS requirements to avoid session expiration.

Azure Data Factory collector: Fixed an issue to accommodate varying data returned from the Azure Data Factory API.

The Power BI Service and Power BI Gov collectors now support harvesting lineage from ODBC data source types. A new parameter --datasource-mapping-file  can be used to provide the information required for harvesting lineage relationships when the data source uses an ODBC connection in Power BI.

The Amazon S3 collector now continues to harvest objects in the bucket when a 403 error is encountered.

The Azure Data Lake Storage Gen2 collector properly handles the scenario involving special characters in the blob name.

The Azure Data Factory collector properly handles a scenario that causes the collector to stop due to the format of information returned from the Azure Data Factory APIs.

BigQuery Collector properly handles a scenario when a table is in a different database from the one being harvested.

The following two new collectors are now available in Private Preview. Please contact your Customer Success Director to get access to these collectors:

The Azure Data factory collector is updated to correctly handle a situation that previously caused the collector to stop, due to the format of the information returned from the ADF APIs.

Amazon S3 collector: The collector now offers the options, --include-object and --exclude-object. These options allow you to select which objects should be included or excluded from the harvesting process.

Databricks collector: The collector now harvests Databricks tags for database, schema, table, view, and column as as key-value pairs. The collector also harvests tags for clusters and jobs, replacing the existing ClusterTag and JobTag resource types.

All collectors: The --dry-run option is now available for all collectors. This option allows you do a test run for the collectors to validate that the collector can authenticate to the specified source system. If specified, the collector does not actually harvest any metadata, but just checks the connection parameters provided by the user and reports success or failure at connecting.

Teradata collector: The collector is updated to correctly parse view SQL syntax for extracting lineage metadata. It also now includes improved logging of any errors encountered during lineage harvesting.

BigQuery collector: The collector now properly handles fully qualified table names that include dashes (-).

Snowflake, Teradata and Netezza collectors: In the harvested metadata, the owner of resources are now correctly referenced as owner objects. Earlier they were referenced as string text.

The Teradata collector now correctly manages variations in database cases within SQL statements while gathering lineage metadata.

The Tableau collector now captures all sub-projects when you specify certain projects to catalog. Additionally, it enables users to exclude specific projects using the --tableau-exclude-project parameter. Any sub-projects under an excluded project are also automatically excluded.

The Information Schema Catalog Collector now collects descriptions from both tables and columns, if they are present in the source.

The Snowflake collector now harvests comments from Snowflake databases, schemas, and views (as resource description).

The Teradata collector has been enhanced to better parse view SQL definitions that use specific Teradata syntax elements, particularly when extracting lineage from views.

BigQuery collector:

Netezza collector: A new and improved collector is now available for Netezza.

Oracle collector: The collector now harvest definitions for view, function and stored procedure.

The following collectors now harvest all databases in a single collector run when the --database parameter is not specified. 

The collectors also support a new parameter --exclude-database to exclude specific databases from metadata collection:

Databricks collector: The collector properly handles malformed task responses.

Power BI collector: The collector properly handles harvesting lineage relationships from Power BI data sources when parameters are used in place of the Snowflake Warehouse value.

For the following collectors, the behavior of the --include-information-schema option is changed. Now, if you use this option in the command without the --all-schemas option, the system will generate a warning to alert you about the missing parameter.

Fixed an issue that was causing database collectors to run into error state.

Azure Data Lake Storage Gen2 collector: Fixed an issue that previously prevented the collector from running successfully on machines using amd64 processor.

Microsoft SQL Server collector now properly harvests views from Azure Synapse Analytics.

Tableau collector: The collector is updated to properly harvest usage data in newer versions of Tableau Server.

Azure Data Lake Storage Gen2 Collector: Fixed an authentication issue in the collector that resulted in failures to initialize a channel.

Snowflake collector: The collector now properly harvests lineage between function and source table if the source table is in the cataloged schema.

All collectors: In addition to being available as Docker Images, collectors are now also accessible as JAR files. Follow these instructions to run collectors using JAR files.

The following collectors now harvest all versions of overloaded function and stored procedure resources, each as its own resource:

Teradata and MySQL collectors: The following schema options have been removed for these collectors: --all-schemas, --include-information-schema, and --schema.

The Snowflake collector now harvests Data Metric Functions, their associations to tables and observed metrics.

Snowflake collector harvests allowed tag values from Snowflake.

Oracle collector properly harvests Column descriptions from Oracle Data Dictionary tables.

The latest tag for docker images has been removed and is not available for use going forward.

What does this change mean for users using the latest tag?

Athena, Snowflake, SQL Server, DB2 collectors now harvest basic metadata for materialized views (name, description if available).

The Postgres collector now collector harvests materialized view with name, description, and view SQL definition (DDL) and column-level lineage.

All collectors: Environment variables referenced in collector config (YAML) files can now have values containing backslashes and dollar signs.

The Snowflake collector now harvests the External URL for Snowsight for tables and views.

The dbt Cloud collector now includes --dbt-cloud-host option to enable interaction with dbt static access URLs.

Databricks collector: Addressed an issue related to correctly forming IRIs for tables under certain circumstances. This was previously causing duplicate tables and databases to be cataloged and non-existent tables to be referenced by columns.

The Tableau collector now properly handles a scenario when the Tableau instance has no databases defined.

dbt Core and dbt Cloud collectors now catalog the dbt product version.

Tableau collector properly handles columns with missing names.

Monte Carlo collector:

Teradata collector: Information was missing while harvesting funtions from Teradata.

dbt Cloud and dbt Core collectors: Information was missing while harvesting test results from dbt Cloud and dbt Core.

The following two new collectors are now available:

The following collectors now harvest Schema resources from the source:

dbt Cloud and dbt Core collectors now harvest following additional metadata: test results (failed, warning, success), last test run timestamp, test name, test arguments and type of dbt test.

Teradata collector:  Information was missing while harvesting triggers from Teradata.

dbt cloud and dbt core collectors now harvests metadata for Columns defined within Models and Sources

The Power BI collector now automatically filter out workspaces named My workspace or PersonalWorkspace <User> when the --all-workspaces-and-apps parameter is used. However, if you wish to include these workspaces in the catalog, you can use the --include-user-workspace option.

Monte Carlo collector: The Monte Carlo collector is enhanced to automatically retry harvesting from Monte Carlo in case of API failure.

All collectors: If errors occur while running the collectors using the YAML file, the collectors will now return a not successful exit status.

The Monte Carlo collector supports harvesting resources from Monte Carlo when the monitored target database is Databricks.

Monte Carlo collector supports new warehouse types in the latest Monte Carlo GraphQL APIs.

dbt cloud and dbt core collectors support versions dbt 1.5.0, 1.6.0, 1.7.0.

Log files for collectors: All collectors now compress log files prior to upload.

Power BI, Sigma, and Thoughtspot collectors: OAuth tokens for the Power BI, Sigma, and Thoughtspot collectors are properly refreshed when they expire.

A new collector is now available for Teradata.

The Databricks collector stops abruptly due to lack of permission on the referenced resources.

Databricks collector: Additional lineage and object metadata are now harvested for database objects via Unity Catalog.

Log messages improvements: Improved logs and guidance is available for situations where SSL certificate problems occur during metadata harvesting from sources secured with self-signed certificates.

Monte Carlo collector: Database Views are now accurately differentiated from Database Tables.

The Manta collector now supports Manta versions R41 and R42.

Manta collector has improved log messages.

Amazon S3, AWS Athena, and AWS Glue collectors now support authentication using AWS config files that reference the credentials on an Amazon EC2 instance profile. For details see the AWS documentation.

Amazon S3 collector: When the collector is run in dry run mode (with the --dry-run parameter), it now also checks the credentials and lists the buckets found.

Monte Carlo Collector:

All collectors:

Oracle collector: The collector gracefully handles exceptions during column harvesting.

Log messages improvements: Collector log messages now include both date and time.

Tableau Collector now harvests column identifiers for column resources.

Confluent Platform Collector now properly handles --include-topic and --exclude-topic options.

Monte Carlo, ThoughtSpot, and InfluxDB Collectors now produce only one catalog record for each resource.

Confluent Cloud collector now support topic filtering using regular expressions or exact topic names using two new parameters: --include-topic and --exclude-topic.

Confluent Platform collector

Databricks collector properly harvests column statistics from columns containing dash characters.

Amazon S3 collector now harvests resources up to and including the maximum count specified by the user or the 10,000 default limit.

dbt Cloud collector handles scenarios where dbt Cloud runs contain more than the current dbt Cloud limit of 1000 artifacts.

Databricks collector properly harvests column statistics from tables with no columns.

MS SQL Server collector properly handles case sensitivity of SQL keywords when parsing lineage.

The following two new collectors are now available:

Power BI and power BI Gov Collectors: The following parameters for these two collectors now support regular expressions: --exclude-workspace --include-workspace

The following JDBC collectors now test for connection status before executing queries. If a connection is closed by the database, the collector detects this condition and re-opens the connection:

Databricks, DB2, MySQL, PostgreSQL, Redshift, Snowflake, MS SQL Server, Azure Synapse Analytics collectors

Databricks collector now properly handles non-alphanumeric characters in object names.

Amazon S3 collector now allows users to filter by bucket with --include-bucket and --exclude-bucket options.

Kafka - Confluent Platform Collector now supports SASL/SCRAM-SHA-512 authentication. If you want to use this authentication, use the --kafka-cluster-sasl-type parameter while running the collector.

New parameters for API retries when API calls fail: The following collectors now support two new parameters ( --api-max-retries and --api-retry-delay) which allow users to specify the maximum number of times the collector will try to reconnect when the API call to the data source fails.

dbt Cloud Collector now properly selects the job that the user specified while running the collector.

The AWS Glue collector properly handles AWS Glue Catalog instances with more than 100 databases.

AWS Glue collector has improved logging to help with troubleshooting of access and permissions issues.

DBT Cloud collector: The following enhacements are made to the collector.

Snowflake and SQL Server Collectors now harvest column-level lineage from Stored Procedures.

Logging improvements:

A new collector is now available for SQL Server Reporting Services.

Databricks collector: Update the collector to retry after a pause when the Databricks API responds with too many requests.

All database collectors: Optimized the database collectors to reuse database connections where possible.

All collectors: Fixed an issue that prevented the user from passing command-line options containing spaces, when running the collectors using the docker container.

Databricks Collector The collector now allows users to use Personal Access Token without specifying username/password for authentication.

Power BI Gov collector: The --include-user-workspace parameter is removed from the collector CLI options.