About the Databricks Publisher automation
Important
This is a Beta feature and is not generally available to all customers at this time. Please contact your Customer Success Director to find out more details.
This automation is available only for customers that have purchased the Data Governance Premium tier.
Configure the automation to manage Databricks comments in data.world. This automation allows you to update comments on Databricks column and table resources and synchronize these updates back to Databricks either automatically or manually using a simple button click.
Important things to know
What resources can be updated?
Only Comments for Tables and Columns sourced from Databricks can be updated in data.world and published back to Databricks. These comments are set as descriptions in data.world.
Can I sync these changes automatically to Databricks?
Yes, when you configure the automation, you have the option to automatically sync changes to Databricks when they are saved in data.world. Alternatively, you can provide users with a Publish Metadata to Databricks button to manually sync the changes to Databricks. When automatic updates are enabled, any changes made to the Description field from the UI (on individual resource pages or through the bulk update/upload flow) will be synced automatically.
How quickly do changes sync to Databricks?
Changes usually sync to Databricks immediately upon refreshing.
What is the Source of truth for comments?
Upon enabling the Databricks Publisher automation, data.world becomes the Source of Truth for table and column descriptions/comments. Users setting up the automation implicitly accept this premise. Adding a description in data.world and subsequently removing it in Databricks will not remove it from data.world—even after a more recent collector run. Therefore, descriptions/comments should be applied or removed in data.world to ensure they are correctly reflected in Databricks.
Are users notified when they update the comments for tables and columns in data.world?
Yes, users are notified when they update the comments/descriptions for tables and columns in data.world. The user updating the comments/descriptions receives notification emails when the comments/descriptions is successfully updated in Databricks or if an error occurs during the update.
Important
These notification emails are only sent when the user has the Confirmation notifications enabled.
Can I setup multiple instances of Databricks Publisher automations?
No, only one Databricks Publisher automation should be set up per organization.