Catalog Toolkit release notes

• dbt Core and dbt Cloud collectors:  Updated the dbt Source to Database Table relationship to align with how these relationships are now gathered by the dbt collectors. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Infor ION collector: Added support for the Infor ION collector. Follow these steps to start using the new collector configurations through CTK.

• Power BI Service and Power BI Gov collectors: Power BI Datasets are relabeled as Power BI Semantic Models to align with Power BI's terminology. . Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Salesforce collector: Added support for displaying Salesforce summary fields and associated metadata. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Catalog configuration improvements:

  • Resouce icons: All data.world hosted icons are now created by default, simplifying the process of assigning custom icons to custom resource types.

  • Read-only configurations: Updated edit hints for read-only configurations to clarify their applicability to both the UI and API.

  • Managing deprecated configurations: Added a default Deprecated MDP collection to simplify the deprecation of specific metadata profile configurations. Refer to the documentation for more details.

  • Metadata fields: Moved the External access configuration field on Metadata fields to Technical details section. This setting is used for creating API Bindings for deprecated legacy API endpoints and is not required for current API endpoints. It remains accessible for those still using legacy endpoints.

• Snowflake collector: Optimized the Snowflake Tags build query to eliminate duplication and resolve an issue with some resources being incorrectly marked as read-only in certain scenarios.

• Azure Data Lake Storage collector: Added the HNS Enabled metadata field to Azure Storage Account resource. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Power BI Service and Power BI Gov collectors: Added Power BI Datasource to Database relationship. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Snowflake Tags: Optimize the Snowflake Tags build query to reduce duplication and file size.

• Tableau collector: Corrected a typo causing Tableau Dashboard to Tableau Field relationships to not be displayed properly. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Alteryx collector: Introduced MDP Module for the Alteryx collector. Follow these steps to start using the new collector configurations through CTK.

• BigQuery collector: Added BigQuery Table and BigQuery View resource types to the BigQuery MDP Module. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• MongoDB collector: Introduced MDP Module for the MongoDB collector. Follow these steps to start using the new collector configurations through CTK.

• Select options configuration for custom fields: Enhanced flexibility in naming column headers when configuring select field options. You can now use variations such as Level 1, level 1, or level_1 in the configuration Excel sheet.

• Grafana collector: Updated the Grafana Panes to Grafana Dashboard relationship to use dct:isPartOf . Follow these steps to start using the new collector configurations through CTK.

• Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Updated Stored Procedure and Function to Schema relationship to use dct:isPartOf. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• ADF and ADLS Collectors: Removed support for deprecated versions of ADF and ADLS collectors. Please update your collectors to a version newer than Release 2.202 for these collectors if you have not already done so. Also, follow these steps to start using the latest collector configurations through CTK.

New features and changes

• Google BigQuery collector: Added Nullable and Column Size metadata fields to the BigQuery MDP module. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Snowflake, Microsoft SQL Server, MySQL, PostgreSQL, Redshift, DB2, Databricks, Tteradata, and Oracle collectors: Updated metadata fields for string sampling metadata for data profiling. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• MS SQL, Netezza, Oracle, Snowflake, Teradata collectors: Added fields for extended metadata. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Search cards: Added support for adding custom metadata fields to search cards. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• QlikSense Cloud collector: Introduced MDP Module for the the new QlikSense Cloud collector. Follow these steps to start using the new collector configurations through CTK.

• Informatica CDI collector: Introduced MDP Module for the Informatica collector. Follow these steps to start using the new collector configurations through CTK.

• Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: Added a Decimal Digits field for database columns.

• Added SPARQL queries to export metadata fields and relationship definitions from MDP Modules.

• Resource Type and Collection Type descriptions from Catalog Configuration are now displayed in the Create Resources modal in catalog organizations.

• Fixed SQL queries for Top 15 Most Viewed Resources, Resource Views, Edits, and Creates by Week, and User Engagements in Platform Analysis.

• Power BI and Power BI Gov collectors: Added new relationships from app report to workspace report. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• All collectors: The Collected Via metadata field is removed, as this information is now displayed in the platform managed Source System field. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Oracle collector: Fixed an error in Oracle build query that was causing sync errors.

• MySQL collector: Added MDP Module for the MySQL collector. Follow these steps to start using the new collector configurations through CTK.

• Salesforce collector: Released a major update to the Salesforce MDP Module to support the enhanced Salesforce collector, which now harvests Salesforce objects. Ensure you update your catalog organization's Metadata Profile to incorporate these changes.

• Added functionality to configure the Source System field as read-only. This update is a precursor to the upcoming release of the platform-managed Source System field.

• Oracle collector: Support added for Package Specification and Package Body Definition fields.

• Azure Data Lake Storage Gen2 collector: Replace redundant permission-related relationships with a single hasPermission relationship.

• ThoughtSpot collector: Removed fields from the MDP that were not in use.

• PowerBI collector: Added support for dataset table expressions and descriptions for workspaces, apps, and datasets.

• Databricks collector: Added suport for Databricks external locations.

• Browse card: Fixed SHACL validation errors when configuring clickable section headers.

• Power BI collector: Fixed the relationship configuration between Power BI Measure to Power BI Table.

• All collectors: Fixed an error that caused source-modules.ttl file to fail to sync in certain situations.

• Added support for custom Source System select options. This update is a precursor to the upcoming release of the platform-managed Source System field.

• Amazon QuickSight collector: Corrected an issue where some metadata fields in the QuickSight MDP Module were not properly populated.

• Power BI collector: Addressed a markdown rendering issue in the Power BI connection details field.

• data.world platform metrics: Fixed an issue where the data.world Metrics dataset was not correctly linked to the data.world Platform Analysis project in some instances, causing sync errors.

• Support added for SQL Server Integration Services (SSIS) collector.

• Azure data factory collector: Support added to display parameters even when the value is empty.

• Power BI and Power BI collectors: Support added for Power BI Measures and Report Type.

• Snowflake, Redshift, Databricks, Denodo, Oracle, PostgreSQL, Teradata, MySQL, Db2, Netezza, SQL Server collectors: The collectors now support Relationships between Database View and Database Column added for Filtered By, Aggregated By, and Ordered By relationships.

• Catalog Configuration Update:

  • Added an Other Modules select field to Catalog Configuration for configuring non-source-specific MDP Modules.

  • Moved the Column Statistics and Sensitive Data Discovery options to the new Other Modules field.

• Browse card configuration:

  • Resolved an issue where multiple items sharing the same title caused display errors and issues with the SHACL validation report.

  • Addressed a problem where spaces in file names used for the Browse Card caused the build query to timeout.

• Power BI collector:

  • The collector now supports Measures and Report Type.

  • The connection details are now visible on the Power BI Data Sources page.

• Collections: The Bulk Collection Import feature has been updated to allow the addition of descriptions and statuses to existing collections.

• Power BI Report Server collector: Errors observed while loading collector output to the intended organization are now fixed.

• Oracle collector now supports Materialized Views.

• Snowflake collector now supports additional metadata and Materialized Views.

• BigQuery collector: The BigQuery labels are now shown correctly on the tagged resources.

• All collectors: The Catalog Configuration ddw-catalog is now automatically synchronized following the deployment of CTK updates. Updates of the Source MDP module will appear in the catalog configuration without the need to manually sync the source-modules.ttl file. However, to apply updates to Metadata profile files will still require manual synchronization in the Catalog organizations.

• Tableau collector: Created and Modified timestamps added to Tableau Dashboards, Tableau Workbooks, and Tableau Views.

• All collectors: Failures observed while syncing source metadata files are now fixed. This issue was observed when there were multiple DWCC output files in a source dataset.

• Resource types: If a resource type with associated metadata fields is deleted and then recreated with the same name, the previously linked metadata fields were displayed on the newly created type.