Make flat fielding step prior to normalization and before generating frames

[pranathivemuri]

flat fielding should be performed on raw data before normalization. so if possible we should see if we can make flat fielding available outside of pre-processing. on another end, it is something that should be corrected immediately after acquisition, so this might not be an issue to address immediately. but again documenting discussion with @smguo

[jennyfolkesson]

Hi @smguo and @mattersoflight,

I'm looking into the zscore normalization issue now to make sure that flatfield correction is happening before zscore parameters are computed.

I see you've added the intensity computations to the generate_meta CLI - is it safe to assume that generate_meta is now always run before preprocess_script? If so, it might make sense to "bake in" the generate_meta features into preprocess_script and then move the intensity computations to after flatfield computations. Or would you rather design it so that you run multiple CLIs one after another? I.e. run a flatfield correction CLI and then run generate_meta on these flatfield corrected images, then preprocess_script?

In the future with zarr support you may not want to save separate tiles but read chunks of the zarr file instead so it might make sense to save flatfield corrected images separately from the originals so that you only have to perform flatfield correction once. That's a separate issue but worth keeping in mind when designing this part.

I'm also wondering why you can only normalize if you have a mask directory in compute_zscore_params in meta_utils? Couldn't you also have an option where you use all metadata if there's no fg_frac (foreground fraction?) or am I missing something?

Let me know how you'd like to proceed with one vs. multiple CLIs and I'll start working on the solution. Thanks!

[smguo]

Hi @jennyfolkesson I think it makes sense to combine generate_meta CLI with preprocess_script, so the new precessing steps could be: generate_meta -> flatfield correction -> intensity computations -> current preprocessing steps. Flatfield can update the file paths in meta csv to point to the corrected images. But I'm fine with either design.

And yes the normalization parameters could be computed without masks but it's not ideal because the images with mostly background would bias normalization parameters. It's not that big of an issue for cell culture data but become really bad for large tissue images where a lot images are empty. And since we're already generating masks for filtering tiles so I made masks a requirement for normalization parameters as well. But we can make it optional.

[ziw-liu]

Flat-fielding is no longer within the planned features.