Skip to content

Quick Start

I. Basic workflow

  1. Create a connection.
  2. Create an integration flow.
  3. Configure the source and target nodes in the flow.
  4. Run the flow to verify it.
  5. Create a transformation flow if needed.
  6. Check the execution status in monitoring.
  7. Create a flow chain and add the created flow to it.
  8. Schedule the chain execution.

II. ELT from PostgreSQL

Integration flow

Full mode

The target table must be created in advance and must contain the technical columns tech_request_id and tech_batch_id.

tech_request_id timestamptz NOT NULL,
tech_batch_id int2 NOT NULL
  1. Create a connection.
  2. Create a new integration flow.

  3. Source node:

    • select a connection from the dropdown list;
    • specify the source schema and table;
    • load the table fields using the button — data contract;
    • keep the mode set to full;
    • click Apply;

  4. Target node:

    • select a connection from the dropdown list;
    • specify the target schema and table;
    • click Apply;
  5. Save the flow.

The flow has been created. Next, run the flow. Or add it to a chain and run the chain.

Delta mode

The target table must be created in advance and must contain the technical columns tech_request_id and tech_batch_id.

tech_request_id timestamptz NOT NULL,
tech_batch_id int2 NOT NULL

Additional steps for full mode.

  1. Source node:
    • set the mode to delta;
    • specify the delta column;
    • specify the delta column type;
    • specify the timezone for the delta column (required if the column type has no timezone);
    • specify the initial date of the delta column from which data will be extracted;
    • specify the batch size by the delta column;
    • click Apply;

Transformation flow

This type of flow in delta mode loads data by batches that were created during the integration stage. In full mode, each flow run extracts all batches from the source.

Flow creation steps:

  1. Create a new transformation flow.
  2. Source node:
    • select a connection from the dropdown list;
    • specify the source schema and table;
    • select the mode, usually delta;
    • click Apply;

  3. Transformation node:

    • if no transformation is needed, write the simplest SQL: SELECT * FROM public.sales WHERE {where_cond_source}
    • click Apply;

  4. Target node:

    • select a connection from the dropdown list;
    • specify the target schema and table;
    • load the table fields using the button — data contract;
    • specify the target database version;
    • click Apply;
  5. Save the flow.