Mirth Connect - Source Control, CI/CD, deployment

[jonbartels]

This has been a popular topic in Slack. It also came up on the MC dev call in August.

Slack only keeps about a month of history so we're losing a lot of chatter on the issue.

I am creating this discussion as a reference to capture the questions or ideas from Slack for how to integrate MC with:

• source control
• CI/CD tools
• automated testing tools
• deployment tools

My take - the tools and parts are all there but they haven't been stitched together in a way that works and makes sense. In my experience it's driven by human processes with some helping tools, but not fully automated yet.

Quick resources from Slack that I could find in Slack history:

• https://github.com/kayyagari/git-ext
• https://forums.mirthproject.io/forum/mirth-connect/support/18382-run-javascript-in-channel-context-outside-of-mirth
• https://github.com/tonygermano/connect-vscode
• https://github.com/vibinernesto/Mirth-Jest-Framework

[AdrianRossouw]

This came up in reference to the last discussion:

https://www.saga-it.com/tech-talk/2019/03/15/mirthsync+installation+and+basic+usage
https://github.com/SagaHealthcareIT/mirthsync

Seems to use an older version of java, but maybe it could be wrapped in a docker container for re-use?

I was also thinking that a script that uses the mirth api's and publish YAML/JSON to them, could be stored in git repository.
Similarly to a helm chart.

[kpalang]

I've been working on a fork of @kayyagari's git-ext to add a remote repository capability and a GUI component to control it. On channel deploy the plugin would also try to push to a specified remote repository.

My vision and need is the following:
On development instance of Mirth I would - well - develop and test a channel. On each deploy the channel would also be pushed to a remote git repository on test branch. Now, should I decide I want to deploy the channel on prod, I would create a PR from test branch to staging. At this point our CD/CD server detects a change on staging branch and creates/updates and deploys the channel on production instance, which in turn triggers the plugin to push the channel to the same repository's live branch.

Here is the initial effort of the script I'm using to deploy the channel to production.

#!/bin/bash

# Try to log in
USERNAME=username
PASSWORD=password
TARGET_URL=https://your-ip-here:8443
COOKIE_FILENAME="cookies.txt"

eat_the_cookies() {
    echo "Eating cookies.."
    if [ -f "$COOKIE_FILENAME" ]; then
        rm -f "$COOKIE_FILENAME"
        
        if [ -f "$COOKIE_FILENAME" ]; then
            echo "Cookies still there..."
        else 
            echo "All cookies eaten!"
        fi
    else 
        echo "No cookies to eat! >:("
    fi
}

CONNECT_TIMEOUT=10
LOGIN_RESPONSE=$(
    curl\
    --connect-timeout $CONNECT_TIMEOUT\
    -ks\
    --cookie-jar "$COOKIE_FILENAME"\
    -X POST "$TARGET_URL/api/users/_login"\
    -H "accept: application/xml"\
    -H "Content-Type: application/x-www-form-urlencoded"\
    -d "username=$USERNAME&password=$PASSWORD")

if [ -z "$LOGIN_RESPONSE" ]; then
    echo "Login request timed out"
    echo "exiting..."
    eat_the_cookies
    exit 1
fi

LOGIN_SUCCESS=$(xmllint --xpath "string(//status)" - <<< "$LOGIN_RESPONSE")

# Make sure login works
if [ "$LOGIN_SUCCESS" != "SUCCESS" ]; then
    echo "Login unsuccessful!"
    eat_the_cookies
    echo "exiting..."
    exit 1
fi

# Get changed files
MODIFIED_FILES=$(git diff staging origin/live --name-only)
for channel_id in $MODIFIED_FILES
do
    if [ ! -f "$channel_id" ]; then
        continue
    fi

    channel_data=$(<$channel_id)
  
    # Update channel code
    PUT_RESPONSE=$(
        curl\
        -ks\
        --connect-timeout $CONNECT_TIMEOUT\
        --cookie "$COOKIE_FILENAME"\
        -X PUT "$TARGET_URL/api/channels/$channel_id?override=true"\
        -H "accept: text/plain"\
        -H "Content-Type: application/xml"\
        -d "$channel_data")

    if [ "$PUT_RESPONSE" != "true" ]; then
        echo "Updating channel failed for $channel_id"
        continue
    fi

    # Deploy channel
    DEPLOY_RESPONSE=$(
        curl\
        -ks\
        --cookie "$COOKIE_FILENAME"\
        --connect-timeout $CONNECT_TIMEOUT\
        -X POST "$TARGET_URL/api/channels/$channel_id/_deploy?returnErrors=true&debug=false"\
        -H "accept: application/xml")
done

eat_the_cookies

[kpalang]

My fork lives here: https://github.com/kpalang/git-ext

[amaster507]

I got a project approval today to add some sort of version control to our Mirth Connect Servers. Being a long time web app developer Git was the first tool of choice with maybe some fancy import/export scripts on change. This should be an interesting project to tackle but very much worth the time to save headaches later when someone thinks they are on test and make changes on production that lose the original configuration if the channels are not backed up anywhere.

Here are some tutorials I found on the matter:
https://www.saga-it.com/tech-talk/2019/03/15/mirthsync+installation+and+basic+usage (same as above)
https://www.saga-it.com/tech-talk/2019/03/01/using+mirthsync+with+git (2nd part)

[pacmano1]

Just a reminder that the paid core extension pack does have the channel history plugin to address problems like you mentioned. I wouldn't work without it for the exact reasons you mentioned. I don't intend my comment as anything other than a reference point. Pluggable SCC would be awesome.

Lastly some of the proposed solutions save changes on channel deploy, that is a poor design IMHO, they need to save on channel save and/or allow "save now".

[mkopinsky]

I put together this document a while back for discussion about testing approaches: https://docs.google.com/document/d/1L2ErvbzSgR58g7hE8Betxy0gEgg7bo0avleOu66FmYM/edit

[jonbartels]

https://www.saga-it.com/tech-talk covers the MirthSync tool which recently shipped a new release https://github.com/SagaHealthcareIT/mirthsync/releases/tag/2.1.0

[srphelan]

Very interested in this topic. We currently use Mirth with around 1000 channels. In house, we have developed python scripts/library to push and pull code from a repository utilizing Mirth’s APIs. It requires some manipulation of the XML (ignoring revision number, sort channelids in certain sections, etc …) but we get a pretty reliable diff. In addition we are exploring automated testing/CI.

For templates/functions, I have taken the approach of direction 3 from the above mentioned google doc. Submit a function name, parameters and the expected result. I have a working mockup but needs more development

For channels/CI, we are looking to utilize python’s unittest. It would submit a message depending on the source connector (copy a file to a directory, send a message to TCP/IP port ..) and then lookup the result using the Mirth’s API and check for the appropriate response/transformation.

Would love to hear what has worked well, what hasn’t, and also maybe a way we all could move forward to help the Mirth community as a whole. While we have a platinum license, this (source control/ automated testing/ CI) is a major gap in its offering for enterprise customers.

[pacmano1]

Why so many channels? Curious. Message volume? Channel per client? Not doing multitenant capable channels? Off topic a bit but interested.

[srphelan]

Mostly channels per client. We work with family physician offices and get appointment data (HL7) and clinical data (CCDAs). In addition, we also work with HIE’s/Hospitals to get notifications for the practice’s patients. Volume can vary greatly between channels (small practices vs large practices).

[AdrianRossouw]

then lookup the result using the Mirth’s API and check for the appropriate response/transformation.

Out of curiousity, which API's are you making use of for this.
We want to do something similar for testing.

[pacmano1]

Probably an offline topic, we do lots of practice management systems (20 or so) for hundreds of clients. That means a handful of channels for each EHR type regardless of number of clients. You probably need a fundamental architecture change. Not being critical here, track me down on LinkedIn if you want to chat about it.

LOL that does not change the importance of this topic on GH of course.

[AdrianRossouw]

This is a summary of how we have been building out CI (but not CD)

It's not final, but it's just how far we've gotten.

Deployment Structure

1. We break up specific integrations (like for instance a specific health system) into different repositories.
2. Each integration is in it’s own channel group.
3. We use a .gitignore in that repository to ensure nothing outside of the channel group is committed.
4. Each integration repository is versioned separately, with semver (v1.1, v2.4.3) etc.
5. Each integration repository builds a version artifact from an azure devops pipeline.
6. it is critically important that no environment specific configs end up in the code at this point.
   It’s one of the things we need to review in the PR for.
   We use global channel maps that are populated from keyvault on channel deploy for any variables that need to be present in a specific environment.
7. We have a separate repository for code templates, that also follow the PR workflow that is also versioned.
8. There is a top level “integrations-manifest” repository that acts as a list of which integrations at which version to deploy.
9. This repository is also versioned (ie: integrations-v1.3.0 includes health-system1-v1.2 and health-system2-v3.0)
10. The pipeline on this repository pulls down all the individual listed assets and compiles them into a complete asset for deployment.
11. On sign-off , it will execute mirthsync to push the directory to the server specified, and deploy the channels.

Dev Workflow:

1. a developer works on the dev environment, building their channels and tests it in that environment.
2. When they are ready to push it to the higher environments, they runs a ‘mirthsync’ command to pull down the channels from dev, and place them in the local git repo
3. they make a PR against the dev branch of the repo
4. After code review it is merged into the repo.

Goals

1. Allow us to work on integrations separately, while avoiding impact to other integrations.
2. Have full traceability of what is deployed to what environment at what time.
3. Implement a code review process, so that weird stuff doesn’t get injected into configurations.
4. Build a single asset that is QA'd and then promoted to higher environments.
5. Give us the ability to roll back to previous releases.
6. Avoid giving production access to developers

Challenges

1. Manual intervention is still required to delete channels / code templates.
2. Configmaps are unusable since there can only be one in the system, and it isn't append only, it replaces everything (unlike the rest of the system)
3. Certain areas don't allow environment variables, for instance http basic auth credentials.
4. In general keeping credentials out of builds is a hassle. Constant vigilance is required.
5. Code templates share a namespace, which makes it possible to wipe out each other's code.
   It would be much preferable if you could put code templates into channel groups.

In closing, it kind of works, and all the pieces are there to make it work, but it does feel like you are going against the grain of the tool.

[siddharthsharma9258]

I think your whole Deployment structure relies heavily on the exports that can be performed in Mirth, output files are all XMLs (unless something has recently and dramatically changed in Mirth).
Do you happen to write extractor that can pull out scripts from the channel, which is the actual code yes, and puts it in a git repo?

[AdrianRossouw]

We make use of MirthSync : https://github.com/SagaHealthcareIT/mirthsync

It uses the API to export the objects, and publishes them back again using the API.

[siddharthsharma9258]

Interesting. I did not know about MirthSync.

[bfnoling]

"5. Each integration repository builds a version artifact from an azure devops pipeline."

What is/are the resulting artifact(s) from the pipeline(s)? What does your Azure DevOps YAML look like (broad strokes or as specific as you want to be)?

Thanks in advance.

[AdrianRossouw]

It's basically just the contents of the repository in a build artifact.
i think we may also have some build pre-processing that happens to try to replace out any known credentials in a last attempt.

We also found it easier to pull down individual build artifacts from the final step where we assemble the final directory structure to sync , rather than doing git checkouts.