Prerequisites: Java, Elasticsearch on http:https://localhost:9200

git clone https://gitlab.com/oersi/oersi-etl.git

cd oersi-etl

The ETL workflows are based on Metafacture, see https://metafacture.org

Pass a directory name to run all *.flux workflows in that directory (or pass a single *.flux file), e.g.:

./gradlew run --args 'data/production/openRub'

This will run all *.flux workflows in data/production/openRub.

To use a *.properties file from another directory for workflow variables, pass its location as the second argument:

./gradlew run --args 'data/production/openRub data/production/oersi.properties'

A oersi.properties file in the same location as the *.flux file is picked up automatically.

By default a local oersi-setup with vagrant up is expected:

cd ../oersi-setup ; vagrant up ; cd ../oersi-etl

Run the workflows in data/production/openRub:

./gradlew run --args 'data/production/openRub data/production/oersi.properties'

Check the responses in *-response.json, search in the backend, e.g.:

http:https://192.168.98.115/resources?provider=["OpenRub"]

Run the workflows that write an Elasticsearch bulk file:

./gradlew run --args 'data/experimental'

Index the Elasticsearch bulk file:

curl -s -H "Content-Type: application/x-ndjson" -X POST localhost:9200/_bulk --data-binary "@data/oersi.ndjson"; echo

Query the index:

http:https://localhost:9200/oersi/_search

curl http:https://localhost:9200/oersi/_search | jq

Delete the index:

curl -X DELETE http:https://localhost:9200/oersi; echo

Tests in src/test/java:

./gradlew check

Generate coverage report in build/reports/jacoco/:

./gradlew jacocoTestReport

Generate SONARCLOUD_TOKEN at https://sonarcloud.io/account/security

Set up a ~/.gradle/gradle.properties file:

systemProp.sonar.host.url=https://sonarcloud.io
systemProp.sonar.login=<SONARCLOUD_TOKEN>

Run SonarQube analysis on sonarcloud.io:

./gradlew sonarqube

See results at https://sonarcloud.io/dashboard?id=oersi_oersi-etl