Bnb/dev #95
Bnb/dev #95
Conversation
…_index precomputed to data handler
…allel passes on single nodes.
…e when only 1 flist chunk.
…t ordering. needed to correct for this in the transform + rotate operation.
|
|
||
| Parameters | ||
| ---------- | ||
| feature : str | ||
| Dataset name to collect. | ||
| masked_meta : pd.DataFrame |
There was a problem hiding this comment.
This method docstring is confusing. What is the difference between masked_meta and masked_target_meta? Is the file paths a subset (chunk) of the full list of file paths? Is target_final_meta_file really a file path? (also target_final_meta_file is misnamed).
There was a problem hiding this comment.
Masked meta is for the file list chunk and masked target meta is for the entire file list. Good catch on the misnaming, here it's not a file.
| extract_workers=None, compute_workers=None, load_workers=None, | ||
| norm_workers=None, ti_workers=None): | ||
| norm_workers=None, ti_workers=None, handle_features=None): |
There was a problem hiding this comment.
docstring for handle_features? why provide separately?
There was a problem hiding this comment.
This was to enable precomputing for the forward pass pipeline. When I was getting threads working these repeated calls to xarray caused issues.
Added stats cli and got process pool for passes on single nodes working.