Catalog collector release note

dbt cloud collector is now available. Detailed documentation about the collector is available here.

Snowflake collector: supports profiling for columns with values stored in scientific notations.

Power BI collector: Fixed an issue with tabular files to properly handle invalid paths or http paths.

Fivetran collector: Updated destination identifiers to match the case for currently supported database types. Specifically, this resolves the duplicate Snowflake resource pages issue.

Snowflake collector: Fixed an issue that was causing duplicate snowflake tag-value pairs.

Tableau collector: Updated project filtering to ensure collector harvests calculated fields which are referenced in a sheet but were not created in the sheet.

Amazon S3 Collector: The new Amazon S3 collector harvests metadata about buckets and objects, including the Region, Version State, Size, Last Modified Data, ACL Owner, Grantee and Grant Permission, amongst others. See all the details about this collector in  this documentation.

BigQuery collector enhancements:

Postgres, Snowflake, Redshift, Microsoft SQL Server collectors: When parsing view definition SQL to harvest column-level lineage, the collectors now correctly parse SQL in which tables are fully qualified in the FROM clause but not in the SELECT clause.

Power BI collector: The Power BI collector has changed the URL used as the dwec:externalUrl property from Power BI's embedUrl to Power BI's webUrl, which now allows the user to open the Report, Dashboard, or Dataset in a browser. Additionally, the collector now harvests the embedUrl from Power BI as a separate property, kos:embedUrl.

Snowflake Collector: The collector now handles scenario when Snowflake JDBC driver does not provide valid default values for certain database columns.

Postgres, Snowflake, Redshift, Microsoft SQL Server collectors: Enhancements have been made to parsing of view definition SQL to harvest column-level lineage. We now support joins on named subqueries and correctly handle quoted identifiers.

Snowflake collector: The sampling queries used to calculate the column statistics were failing.

Manta collector: 

JDBC collectors: The description of the --jdbc-property for JDBC collectors is updated for clarity.

The following additional collectors now provide an option to do a test run for the collectors to validate that the collector can authenticate to the specified source system. This is done by adding the --dry-run parameter while running the collector. If specified, the collector does not actually harvest any metadata, but just checks the connection parameters provided by the user and reports success or failure at connecting.

YAML configuration files used to configure collectors can now interpolate system environment variables and Java system properties. For details about using this feature, see this documentation.

Tableau collector: Improved detection of underlying database type when a Tableau data source uses ODBC.

dbt collector: Fixed an issue in the dbt collector that caused coining of IRIs that were inconsistent with IRIs coined by the Snowflake collector, which prevented the linking of database objects between dbt and Snowflake in the catalog. The application now ensures consistency of database object IRIs created by the dbt and Snowflake collectors.

BigQuery collector: The collector now harvests catalog resources representing BigQuery datasets and their associated metadata.

dbt collector:

Snowflake collector:

Tableau collector The help text now includes examples for the --tableau-project and --tableau-exclude parameters.

Power BI collector The help text now includes examples for the --include-workspace and --exclude-workspace parameters.

A new Further Help section is added to the help available for collectors that is accessed using the -H or --help parameters in the command. It now guides users to the collectors help available on the data.world documentation site.

The collectors now emit a globally unique IRI to track collector runs.

Snowflake collector: Column statistics now supports Number data type.

Snowflake collector: Column statistics now supports columns with spaces in names. 

Tableau collector:  The Tableau collector released from version 2.113-2.117 had an issue because of which it was not able to parse GraphQL queries. If you are using collectors between version 2.113-2.117, you must upgrade to 2.118 to be able to use the Tableau collector successfully.

Power BI Collector: The following alternate options are added for some of the command line parameters:

BigQuery collector:

Snowflake collector: Fixed issue for parsing a SQL statement that contained copy grants in Views. This helps improve the column-level lineage harvested by the collector.

The documentation of the jdbc-property property for database collectors is improved to explain how users can specify multiple properties. This change applies to 19 collectors that include this parameter.

A new resource dwec:Source is added to the catalog emitted from database collectors. It is a mechanism that allows users to render specified resource properties as read-only in the data.world catalog UI.

Power BI collector: The collector now has enhanced parsing of power BI transformation expressions. As a result of this change more column-level lineage information is harvested from Power BI.

Snowflake collector: The collector now harvests table usage counts information.

dbt collector: User-defined database attributes are now enabled for the dbt collector to fully mitigate missing or incomplete profiles YAML file when cataloging database objects referenced by dbt.

The Grafana collector is now available as a private beta release for select customers. Please contact data.world if you are interested in using this collector.

The following additional collectors now provide an option to do a test run for the collectors to validate that the collector can authenticate to the specified source system. This is done by adding the --dry-run parameter while running the collector. If specified, the collector does not actually harvest any metadata, but just checks the connection parameters provided by the user and reports success or failure at connecting.

Catalog graphs (.ttl files) that are automatically uploaded to the data.world platform with -u / --upload are now compressed, enabling larger graphs to be uploaded.

Power BI collector: The collector now provides a new option --disable-expression-lineage to skip parsing lineage from the source expressions.

Snowflake collector: The collector has a new ability to harvest table usage and query count. This functionality is enabled by passing --table-usage-collection. It calculates, for each table in the database being harvested, the percentage of tables in the database that have been queried no fewer times than the subject table. The time period over which this analysis is performed is controlled with option --table-usage-lookback-days (that is, the number of days prior to the time when the collector is being run during which queries of each table are tallied), which defaults to a value of 7.

Snowflake collector: Fixed an issue with SQL parsing in Snowflake for windowed aggregate functions.

Power BI collector: Fixed an issue with the Power BI expression parsing related to joins in source expressions.

Looker collector: Fixed an issue in the Looker collector that caused an abnormal termination of the collector run with certain Looker views.

The following collectors now provide an option to do a test run for the collectors to validate that the collector can authenticate to the specified source system. This is done by adding the --dry-run parameter while running the collector. If specified, the collector does not actually harvest any metadata, but just checks the connection parameters provided by the user and reports success or failure at connecting.

Power BI collector: Updated the Power BI collector to harvest metadata for Dataflows.

Databricks collector: Updated the Databricks collector driver version to 2.6.32. Drivers available here

dbt collector: Updated the dbt collector to harvest metas (as key value pairs) for dbt resources.

Added the following options for Snowflake, Redshift, PostgreSql and MS SQL:

data.world now produce images for the arm64 architecture (in addition to amd64). The addition of arm64 means that dwcc images run seamlessly on M1 mac. As a result of this change, from this release onward two hashes are available per release.

Snowflake collector: Snowflake policy fully-qualified names are being written to the title instead of to the description property.

The --post-process-sparql cli option is now available for all other collectors (This feature was previously made available for some collectors in release version 2.107). This option allows the user to pass in a SPARQL query to post-process the catalog graph created by the collector prior to it being written to the filesystem and/or uploaded to the data.world API.

BigQuery collector: The option to use the credential file for BigQuery no longer allows use of -c . It must be specified with --credentialFile.

Added --post-process-sparql cli option to some collectors. This option allows the user to pass in a SPARQL query to post-process the catalog graph created by the collector prior to it being written to the filesystem or uploaded to the data.world API.

Fixed an issue where MS SQL Server database objects referenced from Power BI did not always link to those objects harvested by the SQL Server collector due to mismatched IRIs.

Google BigQuery collector: The collector is updated to coin IRIs for database objects that align with IRIs coined by other collectors.

dbt collector The collector now writes catalog records for each Snowflake tag and policy.

Fixed a defect in which boolean properties that appeared in the global_options section of a dwcc configuration file were not properly recognized.

Power BI collector: The collector now supports the ability to include specific workspaces for cataloging via the parameter --include-workspace. The collector continues to allow exclusion of specific workspaces with --exclude-workspace. Use of --include-workspace takes precedence.

dbt collector: 

Updated the Monte Carlo collector to add a --bigquery-credentials-file option, in order to standardize the option since the dbt collector has a --bigquery-credentials-file option (note the --big-query-credentialFile still exists in Monte Carlo, this is a new alias for the same option).

The Snowflake collector now harvests Tags, Masking Policies, and Row Access Policies, and associate these resources with the database objects to which they apply. There are new CLI options in order to include these: --tag-collection and --policy-collection.

Updated SQL Server collector to harvest intra-database lineage from views.

Updated the log messages for missing files not required for DBT collector to run.

Added table name to warning messages in Tableau collector in addition to the table ID.

Added parent-child relationship between projects in Tableau collector when the parent project is not included in the filtered projects.

Added pagination for certain queries in Tableau to prevent the result hitting the max node limit.

Updated automatic catalog upload functionality to accommodate large catalog graphs.

Updated the Tableau collector to accommodate changes in the Tableau metadata api that were preventing detection of lineage relationships between Tableau fields and underlying database objects.

Updated the AWS Glue collector to handle an error with jobs that have a space or other invalid characters in their paths.

Updated the Databricks collector to include the UserAgentEntry property in the jdbc connection.

The collectors will now emit the Collector version to the logs.

Added a fix to SQL parsing for window aggregate functions (e.g. SUM(X) OVER (PARTITION by Y ORDER BY Z..) )

Added a filtering feature to filter Tableau fields by project in Tableau collector.

Fixed an intermittent authentication issue associated with harvesting metadata from a single site with Tableau collector.

Added a log message for missing job script in AWS Glue collector.

Enhanced harvesting of column-level lineage from database views, including handling SQL SELECT statements missing a FROM clause, and updated list of Snowflake keywords passed to functions.

Fixed the Fivetran collector so that it doesn't produce "blank" nodes (no id or name)

Added a change to use log_level rather than log-level.

Fixed an issue in Power BI with the new metadata API calls.

Added harvesting of column-level lineage in the dbt collector, for dbt projects that target one of the collectors for which intra-database lineage is supported (i.e., Snowflake, Redshift, and PostgreSQL).

PostgreSQL can now be cataloged using either catalog-postgres or catalog-postgresql for the command.

Microsoft SQL Server collector now harvests SQL Server extended properties for databases, schemas, tables, and columns.

Column-level lineage harvesting in the Snowflake, Redshift, and PostgreSQL collectors now properly harvests lineage from views whose sql statements include comments starting with “--“, and also statements with inline subselects.

The dbt collector harvests process (activity) and model (agent) metadata using PROV-O qualified derivations.

Tableau collector now supports multiple --tableau-project options, allowing the user to include multiple projects in the same collector run.

Tableau collector no longer associates Custom SQL Table resources as part of a database.

The Collector no longer includes a bundled jdbc driver for Salesforce. Please contact support@data.world for assistance in obtaining an appropriate JDBC driver.

Enhancements to the Tableau Collector for project filtering and additional logging

Postgress additional triple for Table to Database linking

Includes some internal dependency updates.

Enhancements to the DBT collector to validate the input profile.yaml file

Enhancement to the tableau collector to add a retry if a graphql query fails

Enhancement to the collectors to add a check at the beginning of a collector run to ensure the output directory exists (if the o/-output option is used), which will log an error and stop if the output directory doesn't exist.

Enhancement to the Tableau collector to reauthenticate if the Tableau API reports failed authorization during the collector run.

Released the new Fivetran collector (catalog-fivetran)

Added the parameter --include-information-schema for Snowflake and SQL Server collectors; we no longer catalog the information schema in these collectors when --all-schemas is specified, unless the user also specifies --include-information-schema.

Improved handling of manifest json structures with some nulls in the dbt collector.

Added reporting on user access issues during parsing/resolution for Snowflake collector.

Updated tableau collector to print a clear log message when authentication expires during a collector run

Update tableau collector to allow optimized serialization of API requests under JDK 17

Significant updates to the Power BI collector. As of this release Power BI Collector outputs different classes than the version before it. The collector now emits information about where it is sourcing it's data.

Internal developer and testing improvements

The MANTA collector is now more specific about the concepts that it emits about Informatica PC.

The information schema collector now explicitly supports Oracle.

We have added enhancements to the dbt collector to harvest DBT snapshots and sources.

For collectors that take API endpoint URLs, the data.world Collector will add a trailing slash to the URL if needed and not specified by the user

New command-line option --warehouse available for the Snowflake collector that allows the user to specify which warehouse to use to connect to snowflake.

Added a the data.world Collector collector for dbt - legacy collector still available.

Fixed an issue with auto-uploaded log files, in which not all log messages were being written out.

Looker collector now emits resources for Looker Views and relates the Looker Dimensions and Looker Measures that are configured within those Views.

Improved handling of unexpected database types encountered when cataloging Tableau.

Further improvements in handling of SQL ORDER BY, GROUP BY, WHERE, and HAVING clauses when harvesting intra-database lineage from database views.

Corrected coining of IRIs in catalog graphs emitted by the Tableau collector.

Improved logging in the Tableau collector to detect unknown linked database types.

Improved harvesting of lineage between database views and referenced columns, including support for columns in SQL ORDER BY, HAVING, GROUP BY, and WHERE clauses, and parsing of a wider range of column expressions.

With this release, the data.world Collector now requires Java 17. If you run the collector from within Docker this change will not affect you. If you run the data.world Collector from a .jar file, you will need to upgrade your JRE to 17 to run DWCv2.80 and greater.

Add the parameter --disable lineage-collection to enable users to turn off cataloging lineage for PostgreSQL, Redshift, MS SQL Server, and Snowflake

The Manta collector can now accept OAuth tokens for MANTA authentication (for harvesting metadata from manta version R35 and above)

We have released a new collector powerbigov that only allows tenantid for auth and not user/password and connects to the government powerbi api urls.

Added some small Tableau collector enhancements.

New PowerBiGov collector with specific endpoints for .gov customers. This collector does not accept a username or password.

For PowerBI, a new way to authenticate is available. A user can now enter a tenant ID with a client id and a client secret to authenticate, in addition to using a username and password.

For both PowerBi and PowerBiGov, when using the tenantid, secret and client id authentication method, this collector no longer emits information about PowerBI Apps.

Fixes an issue with the Snowflake collector that prevented the bundled jdbc driver from being found. Any users working with the data.world Collector 2.76 should update.

The Snowflake collector harvests some intra-database lineage information from database views.

Improved the host mapping in the Manta collector.

the Manta collector now emits resources for file sources and targets and their directory structure. It also emits sources and targets as files.

The Tableau collector now emits “column-level lineage”:

Improved styling of the data.world Collector command-line errors

Updated command-line options for Datakin and Marquez.

Permalinks to Looker explores included via externalUrl

Improvements to datakin/marquez collectors

Tableau collector now emits resources for Tableau Projects, allowing us to establish full relationships between projects and the workbooks and views that they contain

Monte Carlo data collector now emits data quality information using enhanced dwec ontology concepts

Looker collector now emits descriptions for measures and dimensions

MANTA collector now emits Snowflake resources found in MANTA scans

[BUGFIX] Invalid prefix when using --base option

Update the data.world Collector transform to add catalog events to specific collectors

Added a Snowflake Sensitive Data Discovery collector

Sync CLI options between collector types

Validation of CLI options for the data.world Collector

Improvements to the the data.world Collector CLI

Update the MonteCarlo Collector to use the new Data Quality Ontology

Bug fix for Marquez collector

Bug fix for Manta collector

Preview images for Tableau assets are now harvested much more efficiently, and the resulting image data in the catalog graph are much smaller, reducing catalog harvest run time and enabling image objects to remain within platform constraints during ingest.

Fix for unexpected column type errors in BigQuery collector

Created new collectors for Marquez and Datakin

Added schema information to the Tableau collector outputs

The Tableau collector now accepts Tableau “Personal Access Tokens” for authentication, via new cli options --tableau-pat-name and --tableau-pat-secret.

Fixed an issue with mis-identification of views as tables in BigQuery.

Fixed an issue with server name identification in Manta.

With this release the Athena collector is no longer a JDBC collector--we harvest metadata by accessing the Athena API directly, rather than going through a JDBC driver. This means that it is no longer necessary to provide a JDBC driver when running the collector.

This release adds a new option for the MANTA collector, --manta-max-parallel-scenarios. Specifying this option and passing an integer value will configure the MANTA API to export the specified number of scenarios in the MANTA graph in parallel. The default value is 4; adjusting this up or down can improve performance.

The Athena JDBC driver is no longer bundled with the data.world Collector as we have discovered that the Athena driver itself has a dependency on a vulnerable version of log4j. Customers that use the the data.world Collector Athena collector will now need to supply their own driver and put it in the jdbc driver directory (as is done with other collectors for which we don’t distribute a driver).

The Oracle collector now supports connecting to the database via SID (instance ID) or Service Name. Service Name is the default. If a connection via SID is desired, pass the SID as the value of the -d/ --database option and add the --oracle-sid-mode option (flag).

The information schema collector has been modified so that the files table_constraints.csv and constraint_column_usage are now optional, not required.

Updated the BigQuery collector to support current Google Cloud API enhancements.

Added a new Information Schema Collector. This collector runs via the {{catalog-information-schema}} command and is notably cataloging four CSV files that are provided to the collector via a {{--csv-file-directory}} parameter rather than connecting to a database. This collector is an option for customers with tricky DB setups that do not allow them to authenticate or establish connections to their DB via our normal the data.world Collector collectors.

Verified the Manta collector works with MANTA R34.

Released the config file command.

Modified the Tableau collector to remove schema and database names from table names.

Updated the BigQuery collector to support cataloging all datasets in a project at once by default, and to be able to use cli options to select specific datasets in a project as well. With this last change, the  --dataset param is no longer required. The help text has been updated with new messaging to reflect these changes.

Added a new collector for Hive (as an alternative to catalog-hive) that uses only the Hive metastore--it does not connect to the Hive server directly. 

The PowerBI collector now harvests workspaces and identifies other assets as being in workspaces

the data.world Collector now emits “catalog events” into the catalog graph. These capture details about the cataloging process itself, including selected configuration options with which the Collector was run, and summary statistics about the catalog. The ingest process will soon extract this information from catalogs at ingest time and send them to segment for downstream analysis.

Fixed an issue in the Tableau collector in which Custom SQL Table objects without an associated database were not handled correctly.

Added new cli options to salesforce collector so that it can handle sandbox environments and custom login domains customers might have.

This release updates the BigQuery collector to handle fields in BigQuery tables for which the BigQuery API returns null type.

Enhancements to the MANTA collector to harvest additional lineage information from MANTA scans (lineage from Informatica PowerCenter in particular)

Tableau collector enhancement to provide a better warning to the user when an obsolete version of the Tableau API is specified

Fixed an issue with the handling of certain fields and database information when the Tableau collector was run with a non-admin credential.

Fixedan issue with the Tableau collector issue to handle views/workbooks that exist outside of a project.

Fixed an issue with the Tableau collector to handle workbooks that exist outside of projects.

Added a collector for Salesforce (catalog-slesforce)

Fixed a IRI mismatch issue for Tableau Collector when running on Tableau instances with a Snowflake datasource.

Made internal improvements to the tableau collector to enable more efficient querying of the Tableau metadata api.

Fixed an issue in the manta collector in which certain missing data in the MANTA lineage graph caused an exception

Fixed an issue in the Reltio collector that occured when a Reltio configuration was missing certain objects.

Added a collector for Databricks (catalog-databricks)

Fixed an issue with creation of resource IRIs by certain collectors when the user chooses version 1 minting

Updated BigQuery collector to enable integration with data.world platform / connection manager

Fixed an issue with the MANTA collector in which certain large MANTA scans caused a numeric overflow during json de-serialization

Updated Reltio collector to include information about survivorship groups in the emitted catalog

This release fixed an issue with JDBC drivers not being loaded by the Athena collector.

Added a collector for Reltio configurations (catalog-reltio).

Fixes an issue with URLs for sheets and dashboards that exist in Tableau Online or in Tableau Server within a site other than the default site.

Additional datetime fields added for Looker objects and typed as xsd:dateTime.

Fixed an issue caused by an undocumented change in Tableau Online’s REST API when using the Tableau collector to harvest metadata from Tableau Online.

The generic jdbc collector harvests database name when cataloging Intersystems Cache databases

Running the Snowflake collector with the -A / --all-schemas option harvests metadata from all available schemas, as with other collectors

Fixed an issue with permalinks to objects in the MANTA collector .

Added support for Looker folders and additional attributes to the Looker metadata collector.

Added the ability to preview images to the Tableau metadata collector.

Fixed an issue with the handling of certain special characters in catalog resource IRIs.

The --schema option for JDBC collectors can now be specified multiple times to enable the cataloging of multiple schemas in a single catalog.

Updated Tableau collector to paginate additional graphql queries to avoid hitting Tableau Metadata API limits.

Updated the Hive2 collector to capture table-level metadata from the hive metastore

Updated the Tableau collector to allow the user to exclude specified Tableau objects from the catalog

Improvements to error messages produced when using a config-file to run the data.world Collector

We disallow running catalog-postgres and catalog-redshift in the same config file as the two collectors use incompatible JDBC drivers

Improved error handling throughout the data.world Collector

Improvements in representation of Tableau data source names in tableau catalogs

Improvements to the MANTA collector

Upgrade of Denodo collector to Denodo 8

Handle edge case of very large field values embedded in manta’s exported artifacts

Support for sites

Handle edge case of stored procedure columns in manta

Enhancements to domo collector output

Testing improvements

A minor tableau collector enhancement

Fix for an issue in the tableau collector in which column fields were sometimes not properly identifying the Tableau Table from which they sourced their data

Improvment to the presentation of domo catalogs in the platform UI.

Changes to the dockerhub repository where we house images containing non-released versions of the data.world Collector. Previously we were calling these “beta” releases; we now call them “release candidates”. The new repository is datadotworld/dwcc-rc and the image tags are x.y-rc-z where x.y is the next expected Collector release, and z is an increment.

The Tableau collector now emits RDF in which the object of `dct:creator` is a `dwec:Agent` instead of a string literal. This means we write additional details about the Tableau account that created the dashboard, via properties of the `dwec:Agent` resource. These details include: account name, account “full name”, and account email address (if they are populated in Tableau).

The PowerBI collector writes resources representing powerbi “data sources” that are now of a PowerBI-specific class, rather than `dwec:DataArtifact`.

It is now possible for users to set the level (severity) of log messages written to the console and log file. By default, we write “info” level messages; users can choose to write only errors (level=“ERROR”), errors+warnings (level=“WARN”), or all messages including debug trace (level=“DEBUG”). This is useful if we want to have customers run the data.world Collector with debug logging turned on, for troubleshooting problems etc.

License information for the data.world Collector is now available as a subcommand of the data.world Collector. To get all licensing information, run the command docker run -it --rm datadotworld/dwcc:X.XX display-license where X.XX is a version of the data.world Collector greater than or equal to 2.24.

Fixes an issue with the Tableau Server collector when cataloging multi-site server instances.

Adds --tableau-site parameter to enable user to restrict cataloging to a single site (not required, by default all sites in the instance are scanned). Value provided to --tableau-site can be a site ID or name.

Adds the parameter --all-databases to the Athena collector so that it can catalog all the databases accessible from the logged-in account.

Fixes some issues with datatypes for dwec:externalUrl predicates.

The Tableau collector formerly had a CLI parameter --tableau-project-id which could be used to catalog only assets in the project with the specified ID. The parameter is now --tableau-project and takes either a project ID or project name

Update to the MANTA collector to accommodate a minor change in the MANTA API with v 1.31. Customers who have updated their MANTA instance to v 1.31+ will want to use the data.world Collector 2.15+.

The Looker collector now works for non-admin Looker users; however, when the data.world Collector is run by a non-admin, the emitted catalog will not contain any information about databases used by Looker analysis assets (access to database information in Looker requires admin permissions).

All JDBC collectors now populate two new properties for dwec:DatabaseColumn: dwec:columnDefaultValue  and dwec:columnIsNullable, which contain the default value for that column in newly inserted rows, and whether the column can be null, respectively. (Note that only some databases/drivers provide this metadata…we put it in the catalog if it’s there).

Upload generated catalogs via the --upload / -U command-line parameters

Upload the the data.world Collector log when uploading generated catalogs with --upload

Fetch an organization's current catalog with the fetch-catalog command

Support for Denodo and Snowflake

Compatibility of JDBC catalogs with tables imported through data.world integrations

Ability to differentiate source information for databases cataloged from localhost

Cataloging of REMARKS fields into dct:descriptio