JSON split batch

[parkinsonr]

Hello,
I might be missing something obvious but I'm trying to take a JSON message that contains multiple trial objects and split each trial into a separate message going to the transformer. With "split batch" enabled, under the Set Data Types the only option to split the data is by writing something in JavaScript. The description in Mirth for this mentions that the script has access to 'reader', a Java BufferedReader.

My question is, how do I call/reference the incoming message in JavaScript, so that I can then parse it. Assuming I can parse it using a forEach. I tried calling 'msg' but at the source connector level 'msg' is not defined yet. I'm just not sure how to call the inbound message.

Attached a sample JSON with just 2 trials. The actual results has around 700. I need each trial under the DATA object to be a separate message.
TrialJSON.txt

Thank you!

[pacmano1]

It is just an object, so you can reference it via msg:

function convertToCSV(jsonData) {
    const header = jsonData.COLUMNS.join(",");
    const rows = jsonData.DATA.map(row => row.join(","));
    return header + "\n" + rows.join("\n");
}

// Convert JSON to CSV
var csvData = convertToCSV(msg);
// csvData will have your JSON as CSV.

Point is you can use array prototype methods to parse the data, you may not need to batch split it.

[parkinsonr]

Apologies if i wasn't clear, this was my objective:
"I'm trying to take a JSON message that contains multiple trial objects and split each trial into a separate message going to the transformer. "

Its just the way our infrastructure is designed. Our client mirth instances are passing to a server mirth instance which is then calling an API to get the data into a database. The JSON file of 700 trial statuses needs broken apart into separate messages to go back to the server. I also need to translate the JSON into an XML message to go back but I can do that once I get the messages separated.

[pacmano1]

What have you tried arleady in the JavaScript spltter code?

[parkinsonr]

Just something simple like this, but since 'msg' is not defined yet. i'm not sure how to reference the inbound message. Or even if returning my 'obj' is even the right way to pass it back to the channel for split.

const batch = msg.DATA.map(data => {
  const obj = {};
  msg.COLUMNS.forEach((column, index) => {
    obj[column] = data[index];
  });
  return obj;
});

[pacmano1]

Your JSON example file is not valid JSON, you have an extra comma at the end, please take the time to make sure sample files you post are correct.

Put this code in the splitter whiich is almost verbatim from a post by @narupley at the forums, please do check there in the future.

var queue = $gc('queue');
if (!queue) {
	queue = new java.util.LinkedList();
	var obj = JSON.parse(org.apache.commons.io.IOUtils.toString(reader));
	for each (row in obj.DATA) {
		queue.add(JSON.stringify(row));
	}
	$gc('queue', queue);
}

var message = queue.poll();
if (!message) {
	globalChannelMap.remove('queue');
}
return message;

[parkinsonr]

THANK YOU. That did work. I might play around with it to try and get it to pass the column names for each value but this was super helpful. I wasn't making sense how to setup the splitter. When i went to the forums page it said everything was moved to GIT so i was just searching GIT, I didnt realize i needed to search forums too.

Sorry for the comma! I just pulled out the first 2 trials instead of pasting the full 700 and i forgot to remove the last comma. Nice catch.

[parkinsonr]

Was able to tweak the code so the columns map to the data objects.

var queue = $gc('queue');
if (!queue) {
	queue = new java.util.LinkedList();
	var obj = JSON.parse(org.apache.commons.io.IOUtils.toString(reader));
	for each (var row in obj.DATA) {
		var rowObj = {};
		for (var i = 0; i < obj.COLUMNS.length; i++) {
			rowObj[obj.COLUMNS[i]] = row[i];
		}
		queue.add(JSON.stringify(rowObj));
	}
	$gc('queue', queue);
}

var message = queue.poll();
if (!message) {
	globalChannelMap.remove('queue');
}
return message;

[tonygermano]

Here's a more modern update to the code in #5830 (comment). It uses the same idea as #5830 (reply in thread), but replaces the forEach with a reduce. You really shouldn't use for each..in with anything except xml (despite many old examples of it being used with arrays.) Additionally, instead of using a java LinkedList as a queue, this uses a normal javascript array as a stack to accomplish the same thing. While a queue polls from the front of the list, the stack pops from the end. Therefore, the array is reversed to maintain the original order of the data.

var stack = $gc('stack')
if (!stack) {
    const doc = JSON.parse(org.apache.commons.io.IOUtils.toString(reader))
    const rowToJSON = row => JSON.stringify(doc.COLUMNS.reduce((obj,colName,i) => (obj[colName] = row[i], obj), {}))
    stack = doc.DATA.map(rowToJSON).reverse()
    $gc('stack', stack)
}

const message = stack.pop() || null
if (!message) {
    globalChannelMap.remove('stack')
}
return message

You didn't mention how you are getting or processing your messages, but be aware that this solution is intended for single-threaded operation.

[pacmano1]

Closing - but @tonygermano 's answer is probably the most intuitive one these days.