eneel is a cli utility for fast extracting and loading data to and from relational databases. The E and L in ELT.

Mostly uses the source databases proprietary tools for extracting and loading data.

Features:

• Fastest way to extract and load large amounts of data between different databases
• Automatic table creation and conversion of datatypes
• Load a query from one database to another
• Loads are executed in parallel
• Incremental loads
• Pretty runtime logging of load progress
• Database logging
• Limit rows load in development
• Configuration with yaml files. Just list the tables you want to replicate
• Configuration files approach makes versioning with git etc easier

Make sure you have the cli-tools for your required databases then:

• With pip:

  pip install eneel

• Or clone the repository, go to the directory and:

  python setup.py install

• With Oracle you need Oracles CLI client SQLplus
• With SQL server you need CLI tool BCP

After installation an example connections.yml file will be in your home directory (~/.eneel). That's where you configure your connection info to your sources and targets.

Next, create a project configuration file in a directory for you EL projects.

Go to the directory with the project configuration file and run eneel with the project file name excluding yml. I.e for a project configuration file as load_from_postgres_prod_to_dw.yml run:

eneel load_from_postgres_prod_to_dw

The output will then be something like below for at successfull run:

Optional parameters:

• --connections: add a path to the connections.yml you would like to use
• --target: specify which target in connections.yml you would like to use. This will be applied on both sources and targets

• Support for BigQuery
• Incremental loads with updates
• Incremental loads with deletes

• Go ahead and report an issue
• Want to help? Pull requests are most welcome!