Skip to main content

Automating collector runs

Important

This topic only applies to on-premise collector runs.

Harvesting metadata from various data sources consistently is critical in maintaining the modern data ecosystem. To automate this task, numerous tools such as Amazon ECS, Azure Pipelines, CircleCI, etc., are available, each equipped with their unique features and capabilities. These tools ensure frequent and regular scheduling of collector runs, resulting in fresh and up-to-date data for your analysis.

Even though a variety of tools can satisfy different needs, in this documentation, we will primarily focus on the operation of two such utilities: Azure Pipelines and CircleCI.

Utilizing these tools, you can create an automated, self-updating system that automatically gathers metadata, saving efforts and increasing efficiency.