Can a File Reader read more than one input file at a time?

[KevinHobbsSeattle]

There is the Max Processing Threads setting but would that mean a File Reader could read more than one file at a time, I don't think so but wanting to make sure.

I'm looking for a way to improve performance of a File Reader when there are many input files sitting in it's input dir.

Any and all suggestions welcome.
Thank you!

[narupley]

That's right, polling connectors like the File Reader would still only read one file at a time. Theoretically you could have other channels sending messages to that File Reader channel and those messages could process at the same time, though that probably wouldn't be a common scenario. You could also have the source queue turned on, meaning the File Reader would read one file at a time and place them into the queue, and then you could have multiple queue threads pulling from that queue to process messages in parallel. But you'd probably only see a performance increase there if the time it takes to process a message is more than the time it takes to read a message from the filesystem. Turning on the source queue does have a performance hit, but if the time it takes to process a message through the channel is significant, then overall it could be a performance gain.

Adding the ability for the File Reader to read multiple files in parallel would be a good enhancement/request.

[KevinHobbsSeattle]

We do have the source queue turned on and that generally works to build up a queue since for the most part the destination takes longer to process the message than it does to read a line from the input file. But, sometimes it gets close or caught up. But, that is interesting, having the source queue turned off may actually increase performance, something to test out, thank you.

When you say "you could have other channels sending messages to that File Reader channel and those messages could process at the same time", could you explain this further? How can a FileReader accept incoming messages when it's just a file reader, reading lines from a single file at a time?

One way I was thinking of was splitting the files up into multiple directories, having multiple channels reading those files and all sending messages, but this would be into a ChannelReader. Maybe that's what you mean here. Yes, that would be a way to blast in lots of messages to keep some dest queue pressure built up. This channel is downloading from an external source, and we're trying to push it as hard as we can to get maximum downloads per second. I guess we could also split those out and have multiple downloaders.

Thank you for your response!
-kevin

[tonygermano]

@KevinHobbsSeattle You can actually route messages via the User API or with a Channel Writer to any channel. It doesn't need to be a Channel Reader. The difference with a Channel Reader is that it doesn't have any mechanism to generate messages on its own, as a listening or polling channel does.

[narupley]

Yep, you could certainly split your files into multiple directories and have multiple File Reader channels simultaneously reading from the various directories. If your files have predictable filenames perhaps with a numeric or timestamp component, you could also still just have the one directory, but have different filename filters across your File Readers. For example one File Reader could be using *0.txt, the next one *1.txt, or with regexes you can get more complicated like .*[0-4]\.txt etc.

Then all of those File Reader channels would use a Channel Writer to send to a single downstream channel. On that downstream channel you would want to keep the source queue off and set the Max Processing Threads accordingly so that it can process multiple messages simultaneously. For example if you have 10 upstream File Reader channels, set the Max Processing Threads to 10 on your downstream processor channel.

If the time it takes to process a message on your downstream processor channel is significant, you can also turn on destination queueing on your Channel Writer destinations for your upstream File Reader channels. If those destination queues build up, you could go even further and increase the Queue Threads on those destination queues, and also increase the Max Processing Threads on your downstream processor channel. For example if you had 2 destination queue threads on 10 File Reader channels, then you'd want to set the Max Processing Threads on your downstream processor channel to 20. Of course that would only be needed if your downstream processor takes a significant amount of time and your destination queues start building up.

[odoodo]

Usually when having a vast amount of files in a directory, the bottle neck in the connector is more reading in the directory content than the files themselves as the scanning of the directory structure takes extremely long. Using a DirectoryStream circumvents this issue as it allows to iterate over the directory content w/o doing a complete scan first.

This might have the potential to improve the file reader performance.

[narupley]

Makes sense, but this may not work with sorting. For example by default the File Reader reads files with oldest modified timestamp first. And that sort can be changed to filename or file size. So if streaming was used I assume sorting would have to be disabled and whatever default sort (if any) that the OS reports to Java from an ls or equivalent would be used.

[odoodo]

That's true, sorting would be lost. Maybe "Streaming" might however be worth to be added as an option.

As the thread opener considered parallel processing as an option, I assumed that sorting is not relevant in this specific situation.