Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TRUNK-6058 : Create CI Process to create liquibase snapshots #4474

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
TRUNK-6058 : added script to generate liquibase snapshots automatically
TRUNK-6058 : kill the process after installation completing

TRUNK-6058 : added logs

TRUNK-6058 : remove drop database process

TRUNK-6058 : delete installation.properties file

TRUNK-6058 : remove unwanted line

TRUNK-6058 : remove unwanted line

TRUNK-6058 : update script

TRUNK-6058 : update script

TRUNK-6058 : update script

TRUNK-6058 : update script

TRUNK-6058 : change project version

TRUNK-6058 : change project version

TRUNK-6058: fixed the new version issue when generating liquibase snapshots

TRUNK-6058: fixed the new version issue when generating liquibase snapshots

TRUNK-6058: added instructions to run the script

TRUNK-6058: update the Read.me file
  • Loading branch information
ManojLL committed Jan 30, 2024
commit 8603b9817d9ecc009bc433d0cc9884658624808e
27 changes: 25 additions & 2 deletions liquibase/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -80,7 +80,7 @@ a **snapshot** generated from OpenMRS 4.7.x.
* `org/openmrs/liquibase/updates/liquibase-update-to-latest-4.8.x.xml` contains database changes introduced by
OpenMRS 4.8.x

### Further Liquibase files
### Further Liquibase files <a name="#Further-Liquibase-files"></a>
The folder `openmrs-core/api/src/main/resources` contains further Liquibase files:

* `liquibase-schema-only.xml` references the latest snapshot file for creating the OpenMRS schema and continues to
Expand Down Expand Up @@ -125,7 +125,30 @@ need to be updated so that they include the change.
The pom file of the openmrs-liquibase module contains a template for generating Liquibase snapshots from an existing
database and applying snapshots to an OpenMRS database.

### How to generate Liquibase snapshots
### How to generate Liquibase snapshots automatically
#### step 1 - checkout to project root folder
cd <some root folder>/openmrs-core
#### step 2 - run the script `openmrs/liquibase/scripts/generate_liquilbase_snapshots.sh`:
Run the following commands to generate the Liquibase snapshots automatically, where username and password refer to a MySQL user:

. liquibase/scripts/generate_liquibase_snapshots.sh <username> <password>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

After running this script, can you list the files which will get created and their output locations?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dkayiwa I updated the read.me file by adding the details of created and updated files, along with their output locations.


After running this script, the following files are generated.

* `org/openmrs/liquibase/snapshots/schema-only/liquibase-schema-only-<snapshot version>.x.xml` defines the OpenMRS schema. This file is a snapshot generated from OpenMRS current version.
* `org/openmrs/liquibase/snapshots/core-data/liquibase-core-data--<snapshot version>.x.xml` defines core data. Again, this file is a snapshot generated from OpenMRS current version
* `org/openmrs/liquibase/updates/liquibase-update-to-latest-<new openmrs version>-x.xml` contains database changes introduced by OpenMRS new version

and also updated following files,

* `openmrs-core/api/src/main/resources/liquibase-schema-only.xml` references the latest snapshot file.
* `openmrs-core/api/src/main/resources/liquibase-core-data.xml` references the latest snapshot file.
* `openmrs-core/api/src/main/resources/liquibase-update-to-latest.xml` references the latest update file.
* `openmrs-core/api/src/main/resources/liquibase-update-to-latest-from-1.9.x.xml` references the latest update file.
* `openmrs-core/api/src/test/java/org/openmrs/util/DatabaseUpdaterDatabaseIT.java` increase the `CHANGE_SET_COUNT_FOR_GREATER_THAN_2_1_X` variable value by one.
* `openmrs-core/api/src/main/java/org/openmrs/liquibase/ChangeLogVersions.java` include the newly created snapshot and update version to the `SNAPSHOT_VERSIONS` and `UPDATED_VERSIONS` lists.

### How to generate Liquibase snapshots manually
#### Step 1 - Drop your local OpenMRS schema
E.g. by running the script `openmrs-core/liquibase/scripts/drop_openmrs_schema.sql`:

Expand Down
2 changes: 1 addition & 1 deletion liquibase/scripts/fix_liquibase_snapshots.sh
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,6 @@
# The updated snapshots are written to the openmrs-core/liquibase/snapshots folder.
#

openmrs_version=2.4.0-SNAPSHOT
openmrs_version=$(grep -m 1 '<version>' pom.xml | sed -n 's/.*<version>\(.*\)<\/version>.*/\1/p')

java -jar ./target/openmrs-liquibase-${openmrs_version}-jar-with-dependencies.jar
100 changes: 100 additions & 0 deletions liquibase/scripts/generate_liquibase_snapshots.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
#!/bin/bash
#
# This Source Code Form is subject to the terms of the Mozilla Public License,
# v. 2.0. If a copy of the MPL was not distributed with this file, You can
# obtain one at http:https://mozilla.org/MPL/2.0/. OpenMRS is also distributed under
# the terms of the Healthcare Disclaimer located at http:https://openmrs.org/license.
#
# Copyright (C) OpenMRS Inc. OpenMRS is a registered trademark and the OpenMRS
# graphic logo is a trademark of OpenMRS Inc.
#

project_version=$(grep -m 1 '<version>' pom.xml | sed -n 's/.*<version>\(.*\)<\/version>.*/\1/p')


function extract_versions() {
openmrs_version=$(echo "$project_version" | sed "s/\([0-9]*\.[0-9]*\).*$/\1/")

# Extract the major and minor version numbers separately
major_version=$(echo "$openmrs_version" | cut -d'.' -f1)
minor_version=$(echo "$openmrs_version" | cut -d'.' -f2)

# Increment the minor version by 1
new_minor_version=$((minor_version + 1))

# Create the new OpenMRS version
new_openmrs_version="${major_version}.${new_minor_version}"

}

function generate_liquibase_snapshots_and_update() {
cd liquibase/
. scripts/create_liquibase_snapshots.sh "${1}" "${2}"
. scripts/fix_liquibase_snapshots.sh
cd ..
}

function move_updated_snapshots() {
extract_versions
mv liquibase/snapshots/liquibase-core-data-UPDATED-SNAPSHOT.xml api/src/main/resources/org/openmrs/liquibase/snapshots/core-data/"liquibase-core-data-$openmrs_version.x.xml"
mv liquibase/snapshots/liquibase-schema-only-UPDATED-SNAPSHOT.xml api/src/main/resources/org/openmrs/liquibase/snapshots/schema-only/"liquibase-schema-only-$openmrs_version.x.xml"
}

function create_new_liquibase_update_file() {
extract_versions
new_liquibase_file="liquibase-update-to-latest-${new_openmrs_version}.x.xml"
cp "api/src/main/resources/liquibase-update-to-latest-template.xml" "api/src/main/resources/org/openmrs/liquibase/updates/$new_liquibase_file"

database_updater_class="api/src/test/java/org/openmrs/util/DatabaseUpdaterDatabaseIT.java"
variable_name="CHANGE_SET_COUNT_FOR_GREATER_THAN_2_1_X"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we explicitly using 2.1.x here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

according to the steps mentioned here, we are copying the liquibase-updates template and rename it to current version. That template file contains change set as below (this include a comment),
<changeSet id="42ddb873-f064-4418-a6c0-6c09426c4832" author="wschlegel"> <comment>Empty change set for integration tests.</comment> </changeSet>

So when adding a new change-set we need to increase the CHANGE_SET_COUNT_FOR_GREATER_THAN_2_1_X value by one.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But the current version is not 2.1.?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when adding new changeset in a new version which is greater than 2.1, we should increase this CHANGE_SET_COUNT_FOR_GREATER_THAN_2_1_X value by the number of newly added changesets.


# Increase the variable value by 1
sed -i "s/\($variable_name *= *\)[0-9]\+/\1$(( $(grep -o "$variable_name *= *[0-9]\+" "$database_updater_class" | awk '{print $NF}') + 1 ))/" "$database_updater_class"

insert_line=" <include file=\"org/openmrs/liquibase/updates/$new_liquibase_file\"/>"
sed -i -e '$!N;$!N;$i\'"$insert_line" "api/src/main/resources/liquibase-update-to-latest-from-1.9.x.xml"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why are we using 1.9.x here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if the liquibase snapshot update version is above 1.9.x , we add that names in this file. This also mentioned in here

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the documentation that you referenced above, isn't 1.9.x just an example. Where the actual value depends on the current version?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The file called liquibase-update-to-latest-from-1.9.x.xml is used for integration tests. When we make a new snapshot, we create a file for the next snapshot version using the format liquibase-update-.x.xml. After creating this new file, we need to add its path to the liquibase-update-to-latest-from-1.9.x.xml file.


}

function update_log_files_with_new_snapshots() {
extract_versions
sed -i 's/\(<include file="org\/openmrs\/liquibase\/snapshots\/schema-only\/liquibase-schema-only-\)\([^"]*\)\.x\.xml"\/>/\1'"$openmrs_version"'\.x.xml"\/>/' "api/src/main/resources/liquibase-schema-only.xml"
sed -i 's/\(<include file="org\/openmrs\/liquibase\/snapshots\/core-data\/liquibase-core-data-\)\([^"]*\)\.x\.xml"\/>/\1'"$openmrs_version"'\.x.xml"\/>/' "api/src/main/resources/liquibase-core-data.xml"
sed -i 's/\(<include file="org\/openmrs\/liquibase\/updates\/liquibase-update-to-latest-\)\([^"]*\)\.x\.xml"\/>/\1'"${new_openmrs_version}"'\.x.xml"\/>/' "api/src/main/resources/liquibase-update-to-latest.xml"
}

function update_snapshot_tests() {
extract_versions
change_log_java_file="api/src/main/java/org/openmrs/liquibase/ChangeLogVersions.java"
sed -i '/private static final List<String> SNAPSHOT_VERSIONS = Arrays.asList(/s/);/,\ "'"${openmrs_version}.x"'"\);/' "$change_log_java_file"
sed -i '/private static final List<String> UPDATE_VERSIONS = Arrays.asList(/s/);/,\ "'"${new_openmrs_version}.x"'"\);/' "$change_log_java_file"
}

function test_liquibase_snapshots() {
cd liquibase/
. scripts/test_liquibase_snapshots.sh "${1}" "${2}"
cd ..
}

function echo_usage() {
echo "usage: . liquibase/scripts/create_liquibase_snapshots.sh <username> <password>"
}



if [ "${1}" == "" ] || [ "${2}" == "" ]; then
echo_usage
else
echo "[INFO] generate liquibase snapshots"
generate_liquibase_snapshots_and_update "${1}" "${2}"
echo "[INFO] move updated snapshot files"
move_updated_snapshots
echo "[INFO] create new liquibase updated snapshots file"
create_new_liquibase_update_file
echo "[INFO] update the log files with new snapshot version"
update_log_files_with_new_snapshots
echo "[INFO] update the tests with new snapshot version"
update_snapshot_tests
echo "[INFO] generate liquibase snapshots successfully"
fi
#
Loading