Gb/bias nc #181
Gb/bias nc #181
Conversation
…taHandler objects. The data is loaded in init and then full object is passed in serial to the run single calculation
…xo source data to low-res target grid instead of doing a k nearest neighbor
…ature for new exo handlers
grantbuster
force-pushed
the
gb/bias_nc
branch
from
6958a47
to
73912b4
Compare
grantbuster
force-pushed
the
gb/bias_nc
branch
from
d3ec5dd
to
ef4c1e7
Compare
…r arg updates, and other PRs
| @@ -275,6 +261,26 @@ def prep_var_lists(self, variables): | |||
| logger.warning(msg) | |||
| warn(msg) | |||
|
|
|||
| @staticmethod | |||
There was a problem hiding this comment.
you went on a clean up spree huh? nice!
There was a problem hiding this comment.
Haha yeah i thought some of the sphinx issues were being caused by these special imports that weren't being installed... That's not the issue but oh well now it's cleaner.
There was a problem hiding this comment.
and it did turn out to be some weird import thing!
There was a problem hiding this comment.
Haha yeah just not the few that i tried 🤦
| @@ -281,7 +281,9 @@ def extract_feature(cls, | |||
| time_slice) | |||
|
|
|||
| else: | |||
| msg = f'{feature} cannot be extracted from source data.' | |||
| available = [str(var) for var in handle.variables] | |||
There was a problem hiding this comment.
I think we can use cls.get_handle_features here, for both nc and h5
| @@ -179,8 +185,7 @@ def __init__(self, | |||
| def source_data(self): | |||
| """Get the 1D array of source data from the exo_source_h5""" | |||
|
|
|||
| def get_cache_file(self, feature, s_enhance, t_enhance, s_agg_factor, | |||
| t_agg_factor): | |||
| def get_cache_file(self, feature, s_enhance, t_enhance, t_agg_factor): | |||
There was a problem hiding this comment.
dont we want the cache identified by spatial agg factor?
There was a problem hiding this comment.
Yeah I actually removed the s_agg_factor from the whole module altogether because for the topo aggregation "factor" it's now determined by the actual pixel mapping.
| self.bias_gid_raster = np.arange(lats.size) | ||
| self.bias_gid_raster = self.bias_gid_raster.reshape(raster_shape) | ||
|
|
||
| self.nn_dist, self.nn_ind = self.bias_tree.query( |
There was a problem hiding this comment.
So it looks like theres no spatial agg being done anymore for bc? I think circular agg is still better than none
There was a problem hiding this comment.
Yeah so there are two options with low->high res mapping:
- Tree(high_res).query(low_res, k=1000)
- Tree(low_res).query(high_res, k=1)
We were previously doing 1 which maps circles of high-res pixels to each low-res centroid which is fine but double counts some stuff in the circle overlaps.
I moved the bias calc and topo exo extraction to 2 which takes every high-res pixel and assigns to nearest low res pixel. This basically makes a sudoku style sub-grid aggregation. The mapping happens in the two places linked below (for bias calc, exo happens somewhere else).
I checked the difference between 1 and 2 in the bias calc and the differences were negligible in the bulk of the extent with differences 1-5% at the borders of the extent.
The biggest caveat here is that the high-resolution must be higher resolution than the low-resolution data for the mapping to work. The main failure mode is if a low-res pixel doesn't get assigned a unique high-res pixel. I put in a method to gap fill this but it's not great.
There was a problem hiding this comment.
Yeah, missed the tree being built on low_res. Good approach!
@bnb32: refactor for sub-grid aggregation for bias calculations, topo extractions, and a feature to allow the baseline input file in bias calculation to be .nc (e.g., ERA)