"source filter" vs "destination filter" and "single destination" vs "multiple destinations"

[JeffBenedict]

I've been searching both forums for conversations regarding filtering. My general take away from the forum topics is to filter as early as possible, if you can, in order to reduce resource usage and disk space.
Has anyone written up a summary guide for what and where filtering should occur for optimal operation of the engine related to different scenarios like Single Destination channels, multi-Destination channels, channel "pass-thru/broker channels" for assistance w/ "parallel processing" (ie.. sending a single inbound ADT to many (30+) destination outbound channels, 10 in each group).

Thanks in advance.

[jonbartels]

Theres no writeup that I know of. Some other places to search would be blogs from Vibin, Apps By Aaron, and MCUG. What forum references did you find already? Does the MC user guide cover it in detail or just a quick summary of what the buttons do instead of a "when and why" guide?

Maybe you could create a Gist https://gist.github.com/ and create a markdown document with what you already know with placeholders for other questions.

A lot of engineers like to use destinationSet.remove calls in the source transformer instead of filtering. This significantly reduces message storage and visual noise because it literally removes the destination from the call stack. A filter runs the filter and still produces a "FILTERED" status and a message stored. This works great for cases where the source message has enough data to make filtering decisions. It does not work for cases where destinations are chained and the filter on a destination depends on data from a previous destination.

If your primary goal is storage management, the pruner is also key. This script is good for Postgres backed MC instances - https://gist.github.com/MichaelLeeHobbs/67980d165fc68880eb2ab283c673244b . It is focused on pruning, but it could also be used to find channels that need filters or destinationSet calls earlier in their processes to avoid passing useless messages downstream.

Please do copy-paste anything I wrote above into any documentation you might create. Its a good idea to consolidate that knowledge into a guide.

[pacmano1]

Not sure there is a guide per se. I would definitenly use destinationSet (i.e. destinationSet.remove([metaDataIdOrConnectorNames]), destinationSet.removeAllExcept([metaDataIdOrConnectorNames]), etc.) filtering for channels with lots of destinations vs filters on those destinations because it reduces storage requirements.

[pacmano1]

jinx @jonbartels . Also using the attachment handler where relevant has a large storage impact.

[jonbartels]

Love you pookums. It's cute how we have the same replies all the time.