Skip to main content

KNOTS: Desktop super connector for many common data sources

KNOTS is a quick and intuitive visual ETL tool that allows you to do complex data replication with ease. Using the visual interface, you can now easily bring your data into data.world with the power of Singer taps and targets. Taps and targets are applications that can be combined together to create simple data pipelines. Taps extract data from a source while targets consume them.

Using KNOTS, you can import data from a number of datastores on an ad-hoc basis or you can export a fully-configured knot and run it with a job scheduler to make sure your data is always up-to-date. With the intuitive interface, you can configure robust data replication processes in minutes, and KNOTS is always free.

With KNOTS and data.world, you can:

  • Leverage ETL without complex manual configuration

  • Quickly bring data from various datastores into data.world

  • Pause and resume replication without any loss of progress

KNOTS is currently only available on MacOS. A Windows version is in the works; if you'd like to be notified when it's complete, please create a new ticket.

Prerequisites

KNOTS depends on Docker being installed and running. Docker is a tool designed to make it easier to create, deploy, and run containers. Containers allow us to package up an application or library with all of its dependencies. Each individual tap and target is packaged into its own container with the correct set of dependencies to ensure they’re all easy to use.

The installer for Docker for Mac is available here.

NOTE: Check Docker file sharing preferences and make sure that "/Users" is a shared directory.

Download KNOTS & Docker.

Before you can start replicating your data, you'll need to download and install KNOTS and Docker:

Download for MacDownload for Windows

Installation instructions for Mac

Installation instructions for Windows

  1. Download the correct package from the newest releases in the KNOTS repository on GitHub (KNOTS is built on top on Singer.io taps & targets).

  2. Download the correct Docker package for your system (you will need to create a free Docker account if you don't already have one).

  3. Install KNOTS and Docker on your computer.

Create your knot.

Now that KNOTS & Docker are installed, you can begin creating your first knot:

  1. Open KNOTS & Docker (make sure Docker is open & running before starting your first knot.)

    When opening KNOTS in Windows, you may receive a popup that states "Windows protected your PC". Click on More info, followed by Run anyway. Docker Tips:

    • On Mac: Check the Docker file sharing preferences to make sure Users is a shared directory.

    • On Windows: The first time you set up a tap, Docker will ask to share the C drive. Accept, and enter your password if prompted.

  2. Select your tap (the source you want to use to replicate data).

    knots_choose-tap.png

  3. Configure your tap. You'll need to enter the following information:

    • Hostname.

      If you are running your database locally, enter host.docker.internal instead of localhost

    • Port.

      Most apps have a default port (5432 for PostgreSQL, 3306 for MySQL).

    • Your database name.

    • Username & password for your database.

    knots_config-tap.png

  4. Select any and all data that you would like to replicate.

    knots_choose-data.png

  5. Select data.world as your target.

    knots_choose-target.png

  6. Configure the data.world target. You'll need to enter the following information:

    • Dataset URL.

      Choose Browse to choose from a list of available datasets.

    • Your API token (from your data.world's KNOTS integration page).

    • Username & password for your database.

    knots_config-target.png

  7. Choose a name for your knot, then choose Save & Run.

While your knot is running, you can cancel and and resume at any time without any loss of progress.

Run with a scheduler

As of today, KNOTS allows you to update the data by manually clicking on the Sync new data action. By making use of the Export action though, and the resources that it provides, it's possible to set up a job that will update the data automatically on a schedule.

The exported package is a ZIP file that includes the tap and target for your knot, as well as their configurations, and a Makefile. You can read more about Makefiles here, but the gist of it is that it contains all the necessary commands to run the knot and keep your data updated. Through the use of a tool like crontab, you can run the knot from your local computer, or it can be set up to be executed from a cloud service like Amazon Web Services or Heroku.

What next?

Here are a few things you can do with KNOTS and data.world:

  • With any saved knot, you can sync new data, sync all data, and edit, export, or delete your knot.

  • Using the Export function, you can set up a job that will update your data automatically on a schedule. Learn more about using a job scheduler here

Support

If you run into issues or have questions, please submit a ticket to the data.world team.

In this section: