File Reader / Writer not processing the full file before writing?

[martinjaehn]

So I created a channel to transform a CSV file from one structure into another.
The source and destination paths are local folders on the Mirth Connect server.
Batch processing is disabled, because the whole message should be written at once and to one file once everything is processed.
To process all the rows/lines, I added a for loop into the transformer.
It takes about 1h to process a file with 10k rows.
I found this a bit too long, so I decided to see what happens if I implement the destination as a JavaScript Writer instead of a File Writer (the source connector is still a File Reader):

// Set serialization and deserialization properties
var serializationProperties = SerializerFactory.getDefaultSerializationProperties("DELIMITED");
serializationProperties.put("columnDelimiter", ";");
var deserializationProperties = SerializerFactory.getDefaultDeserializationProperties("DELIMITED");
deserializationProperties.put("columnDelimiter", ";");

var rawData = connectorMessage.getRawData();
var records = rawData.split("\n");
var outboundData = "";

for (var i = 0; i < records.length; i++) {
	// Serialize record/row to XML
	var msg = new XML(SerializerFactory.getSerializer("DELIMITED", serializationProperties, deserializationProperties).toXML(records[i]));

	// Make changes to the record/row
	msg['row']['column1'] = "";
	msg['row']['column2'] = "";    
	msg['row']['column3'] = "";
	
	// Add record/row to outboundData
	outboundData += SerializerFactory.getSerializer("DELIMITED", serializationProperties, deserializationProperties).fromXML(msg);
}

// Write outboundData to file/destination path
var writer = new java.io.BufferedWriter(new java.io.FileWriter($('destinationPath') + $('originalFilename')));
writer.write(outboundData);
writer.close();

This took about 30 seconds to complete. This kind of indicates to me that Mirth Connect isn't reading and/or writing the full file at once and consequently makes too many calls to the file system. I checked the source code, but couldn't find anything to prove this. Still, there has to be something different to what my JavaScript is doing, because my JavaScript is so much faster.

So what exactly is it, that slows the File Reader / File Writer down so much? Exactly how are files handled by the File Reader / File Writer? Reading line by line? Writing line by line to a temp file, resulting in way too many file system calls and then moving/copying it to the actual destination?

The JavaScript works great, I'm just really confused as to how and why the performance differentiates so much.

[odoodo]

Very true, but not an answer to my question.

You are perfectly right. That's why I deleted my post after I realized that I misread your question ;-)

[martinjaehn]

Ohh that's why I couldn't find it here ^^ was a bit confused about that (first time using discussions).

[tonygermano]

You don't show how this was done when you were still using a File Writer. Were you using the same code with Raw inbound and outbound types for your transformer? Or did you have your types set to Delimited?

e4x gets really slow the more children are added to an element (rows in your case if the type was delimited.)

In your Javascript Writer code you get around this by deserializing and serializing a line at a time and building the result as a string instead of a giant XML object that gets serialized all at once.

I have a feeling if you used the same transformer from the File Writer, and only used the Javascript Writer to write the resulting message to a file, it would take the same amount of time.

[martinjaehn]

I have a feeling if you used the same transformer from the File Writer, and only used the Javascript Writer to write the resulting message to a file, it would take the same amount of time.

I just tested this and this is correct. So it looks like it is e4x getting slow, as you mentioned. Thank you :)