About the Databricks collector
Use this collector to harvest metadata for Databricks resources across the enterprise systems and make it searchable and discoverable in data.world.
Note
The latest version of the Collector is 2.159. To view the release notes for this version and all previous versions, please go here.
What is cataloged
The collector catalogs the following information.
Object | Information cataloged |
---|---|
Columns | Name, Description, JDBC type, Column Type, Is Nullable, Default Value, , Column size, Column index |
Table | Name, Description, Schema, Primary key, Foreign key, Owner, Type, Creation, Last Modified, Location, Provider, Version, Size, File Count, Partition Columns, Properties |
Views | Name, Description, SQL definition |
Schema | Name |
Database | Type, Name, Server, Port, Environment, JDBC URL |
Notebook | Notebook ID, Path, Language Type (SQL, Python, Scala, R) |
Function | Name, Description, Function Type |
Job | Title, Description, Creator, Created At, Job run as, Format, Max Concurrent Runs, Notification On Start, Timeouts (sec), Notification On Success, Schedule, Git Source, Notification on Failure, Job Tags, List of tasks, List of clusters |
Cluster | Name, Description, Node Type ID, Driver Node Type ID, Spark Version, Number of Workers, Autoscale Max Workers, Autoscale Min Workers, AWS Attributes, Cluster Tags |
Task | Task Key, Type of Task (Notebook, dbt, Spark jar, Python script, Python wheel, Pipeline task, SQL), Task timeout, Retry interval, Cluster used by the task, Max retries, Depends on, Libraries, Notifications (On start, On success, On failure), Notebook File Path, Notebook Source, Notebook Parameters, Spark Jar Main Class Name, Spark Jar Parameters, Python Script File path, Python Script Parameters, Spark Submit Parameters, Pipeline ID, Pipeline Full Refresh, Python Wheel Package Name, Python Wheel Entry Point, Python Wheel Parameters, SQL Warehouse, SQL Query ID, SQL Dashboard ID, SQL Alert ID, Dbt Project Directory, Dbt Profiles Directory, Dbt warehouse, Dbt catalog, Dbt schema, Dbt commands |
Relationships between objects
By default, the harvested metadata includes catalog pages for the following resource types. Each catalog page has a relationship to the other related resource types. If the metadata presentation for this data source has been customized with the help of the data.world Solutions team, you may see other resource pages and relationships.
Relationship page | Relationship |
---|---|
Table |
|
Columns |
|
Job |
|
Cluster |
|
Task |
|
Notebook |
|
Folder |
|
Job Tag |
|
Cluster Tag |
|
Lineage for Databricks
The following lineage information is collected by the Databricks collector.
Object | Lineage available |
---|---|
Column in view | The collector identifies the associated column in an upstream view or table for both Hive metastore and Unity Catalog:
|
Notebook | Tasks that reference Notebook. (Only if Databricks Unity Catalog is enabled). |
Table | The collector identifies the upstream and downstream tables along with the intermediate Job. (Only if Databricks Unity Catalog is enabled). |