File Reader / Writer not processing the full file before writing? #4582
-
So I created a channel to transform a CSV file from one structure into another. // Set serialization and deserialization properties
var serializationProperties = SerializerFactory.getDefaultSerializationProperties("DELIMITED");
serializationProperties.put("columnDelimiter", ";");
var deserializationProperties = SerializerFactory.getDefaultDeserializationProperties("DELIMITED");
deserializationProperties.put("columnDelimiter", ";");
var rawData = connectorMessage.getRawData();
var records = rawData.split("\n");
var outboundData = "";
for (var i = 0; i < records.length; i++) {
// Serialize record/row to XML
var msg = new XML(SerializerFactory.getSerializer("DELIMITED", serializationProperties, deserializationProperties).toXML(records[i]));
// Make changes to the record/row
msg['row']['column1'] = "";
msg['row']['column2'] = "";
msg['row']['column3'] = "";
// Add record/row to outboundData
outboundData += SerializerFactory.getSerializer("DELIMITED", serializationProperties, deserializationProperties).fromXML(msg);
}
// Write outboundData to file/destination path
var writer = new java.io.BufferedWriter(new java.io.FileWriter($('destinationPath') + $('originalFilename')));
writer.write(outboundData);
writer.close(); This took about 30 seconds to complete. This kind of indicates to me that Mirth Connect isn't reading and/or writing the full file at once and consequently makes too many calls to the file system. I checked the source code, but couldn't find anything to prove this. Still, there has to be something different to what my JavaScript is doing, because my JavaScript is so much faster. So what exactly is it, that slows the File Reader / File Writer down so much? Exactly how are files handled by the File Reader / File Writer? Reading line by line? Writing line by line to a temp file, resulting in way too many file system calls and then moving/copying it to the actual destination? The JavaScript works great, I'm just really confused as to how and why the performance differentiates so much. |
Beta Was this translation helpful? Give feedback.
Replies: 2 comments 2 replies
-
You are perfectly right. That's why I deleted my post after I realized that I misread your question ;-) |
Beta Was this translation helpful? Give feedback.
-
You don't show how this was done when you were still using a File Writer. Were you using the same code with Raw inbound and outbound types for your transformer? Or did you have your types set to Delimited? e4x gets really slow the more children are added to an element (rows in your case if the type was delimited.) In your Javascript Writer code you get around this by deserializing and serializing a line at a time and building the result as a string instead of a giant XML object that gets serialized all at once. I have a feeling if you used the same transformer from the File Writer, and only used the Javascript Writer to write the resulting message to a file, it would take the same amount of time. |
Beta Was this translation helpful? Give feedback.
You don't show how this was done when you were still using a File Writer. Were you using the same code with Raw inbound and outbound types for your transformer? Or did you have your types set to Delimited?
e4x gets really slow the more children are added to an element (rows in your case if the type was delimited.)
In your Javascript Writer code you get around this by deserializing and serializing a line at a time and building the result as a string instead of a giant XML object that gets serialized all at once.
I have a feeling if you used the same transformer from the File Writer, and only used the Javascript Writer to write the resulting message to a file, it would take the same amount …