Troubleshooting the collectors

If your run of the data.world Collector does not capture everything in the catalog that you think should be there, the first thing to check is the user account you use to connect to your resource to ensure that you can authenticate to the resource outside of the collector and find those objects. For instance, with a database, you should be able to log into the database with a client (preferably a JDBC client like DBeaver) and see the objects. If the objects don't show up there either, it's a permissions issue.

When you run a collector, the output file name is of the form [database name].[collection name].dwec.ttl. The result is that any time the collector is run more than one time against the same database and uploaded to the same collection, the output file will be overwritten. Overwriting the results when cataloging all schemas in a database is fine as the previously produced file is just updated.

However there are instances--e.g., when it is necessary to catalog one schema in a database at a time--where using the same name for the output file results in an overwrite of unique information as opposed to an update. In this case it would be desirable to have unique names for each of the output files before they were uploaded to a collection in the catalog.

Currently the way to achieve uploading of unique files from different schema in the same database is to:

When running the Collector via Docker to catalog large bodies of metadata (e.g., a data source with hundreds or thousands of tables and many thousands of columns), you might exhaust the available memory in the Docker container for the collector process. To address this problem, increase the memory available to Docker. On Windows and MacOS, this is handled via a Docker desktop preference change. If you are running this on a native Linux host, the Docker host and native host are the same (so memory available to Docker is all machine memory). On a Mac, e.g., go to Docker preferences:

And select Resources > Advanced. In this example the memory allowance is set to 2 GB. Increase it to 4 GB by moving the slider for Memory:

You can also increase the memory available to the data.world Collector container by terminating other containers running within the Docker host.

If you are using a collector for which the JDBC driver is bundled in the data.world Collector, you might get a log message (INFO level) upon running the collector noting that no jdbc drivers were found in the directory where the data.world Collector looks for drivers. This message can safely be ignored. This message is not an error message or a sign that anything is wrong--those messages are indicated with WARN or ERROR severity. You do not need to (nor should you) provide your own driver.

All database options passed through the data.world Collector (the name of the database, the name of any schemas, role, user, and password, etc.) are all case-sensitive.