About the Azure Data Factory Collector
Important
The collector can be run in the cloud or on-premise using docker or JAR files.
Azure Data Factory (ADF) empowers users to collect, transform, and relocate data. Use this collector to harvest metadata from ADF, encompassing details on pipelines, datasets, dataflows, linked services, triggers, integration runtimes, and global parameters. Additionally, it gathers lineage information between ADF datasets and between ADF and external sources such as Snowflake.
Note
The latest version of the Collector is 2.239. To view the release notes for this version and all previous versions, please go here.
What is cataloged
The collector catalogs the following information from Azure Data Factory.
Object | Information collected |
---|---|
Factory |
|
Pipeline |
|
Pipeline Activity |
|
Linked Service |
|
Dataset |
|
Dataflow |
|
Trigger |
|
Integration Runtime |
|
Global Parameter |
|
ADF Table |
|
ADF Column |
|
Relationships between objects
By default, the data.world catalog will include catalog pages for the resource types below. Each catalog page will have a relationship to other related resource types. Note that the catalog presentation and relationships are fully configurable, so these will list the default configuration.
Resource page | Relationship |
---|---|
Factory |
|
Pipeline |
|
Activity |
|
Linked Service |
|
Dataset |
|
Dataflow |
|
Integration Runtime |
|
Trigger |
|
Lineage for Azure Data Factory
The following lineage information is collected by the Azure Data Factory collector.
Object | Lineage available |
---|---|
Dataset | The collector identifies the source of the dataset:
|
Supported cross-system lineage
The currently supported data sources for cross-system lineage are:
Snowflake
Databricks
Important
While other data sources are not formally supported, running the collector for those sources may still enable you to view cross-system lineage between Azure Data Factory and these sources.
Authentication supported
Authenticate to Azure Data Factory using Service principal.