Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gb/bc #96

Merged
merged 29 commits into from
Oct 4, 2022
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
efcf50e
added bias module with a place for bias transformation functions and …
grantbuster Sep 20, 2022
115e9c2
feature specific bias correction
grantbuster Sep 20, 2022
c184917
removed warnings about excessive padding - not a bad thing
grantbuster Sep 20, 2022
1d823e3
added a site-by-site linear bias correction calculation method
grantbuster Sep 21, 2022
8b7f25b
bug fix and logging
grantbuster Sep 21, 2022
56148db
bias calc mods and new functions
grantbuster Sep 22, 2022
50c3692
added bias calc cli
grantbuster Sep 22, 2022
2f7d4f7
added bias calc to main cli
grantbuster Sep 22, 2022
c132b81
make bias out dir
grantbuster Sep 22, 2022
915edd5
bug fixes and minor refactor to run on eagle
grantbuster Sep 22, 2022
7b28943
added local linear bias correct to forward pass bc options
grantbuster Sep 22, 2022
0cee671
added option to smooth spatial bias correction factors outside of the…
grantbuster Sep 23, 2022
aa0a040
better enumerated progress logging for fwp
grantbuster Sep 23, 2022
240a0d6
added bias correction option to QA
grantbuster Sep 23, 2022
9ac0905
minor refactor to bias correct u and v instead of windspeed and direc…
grantbuster Sep 23, 2022
d2fb1e2
fixed up the u/v QA with bias correction
grantbuster Sep 27, 2022
01fbeda
added meta data to bc h5 output attrs
grantbuster Sep 27, 2022
b370b9f
more bc convenience functions
grantbuster Sep 28, 2022
24c40b6
added monthly bias correction
grantbuster Sep 28, 2022
0deab95
added montly bias correction data transformation method and integrate…
grantbuster Sep 29, 2022
1f28ccc
fixed collection logic for undefined mask meta variable when file is …
grantbuster Oct 2, 2022
5d93283
added bias correction calc tests
grantbuster Oct 3, 2022
0638d7b
added bias transform calcs
grantbuster Oct 3, 2022
6cc6ced
added fwp+bc integration test
grantbuster Oct 3, 2022
ca24793
added qa+bc integration test
grantbuster Oct 3, 2022
b75b3fc
added version record to bias calc output files and incremented versio…
grantbuster Oct 4, 2022
b0a2c49
simplify qa test and pylint issue
grantbuster Oct 4, 2022
7b9c88f
fixed test on h5 meta attrs dtype and docstrings
grantbuster Oct 4, 2022
2ea15e3
serial data handling for QA+BC bug
grantbuster Oct 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
more bc convenience functions
  • Loading branch information
grantbuster committed Oct 4, 2022
commit b370b9fa1339922342b8173bcb6e0d0fa1986f60
62 changes: 50 additions & 12 deletions sup3r/bias/bias_calc.py
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,27 @@ def get_node_cmd(cls, config):

return cmd.replace('\\', '/')

def get_bias_gid(self, coord):
"""Get the bias gid from a coordinate.

Parameters
----------
coord : tuple
(lat, lon) to get data for.

Returns
-------
bias_gid : int
gid of the data to retrieve in the bias data source raster data.
The gids for this data source are the enumerated indices of the
flattened coordinate array.
d : float
Distance in decimal degrees from coord to bias gid
"""
d, i = self.bias_tree.query(coord)
bias_gid = self.bias_gid_raster.flatten()[i]
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Include a check that the index is in the raster?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

there's no distance threshold on this method so i should always be in the gid raster. I don't include a distance threshold because this is just a convenience method for debugging and investigation, not used in the actual bc module in a pipeline.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There's no default threshold?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nope its infinite haha

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh lol

return bias_gid, d

def get_base_gid(self, bias_gid, knn):
"""Get one or more base gid(s) corresponding to a bias gid.

Expand All @@ -222,15 +243,13 @@ def get_base_gid(self, bias_gid, knn):
dist, base_gid = self.base_tree.query(coord, k=knn)
return dist, base_gid

def get_data_pair(self, bias_gid, knn, daily_avg=True):
def get_data_pair(self, coord, knn, daily_avg=True):
"""Get base and bias data observations based on a single bias gid.

Parameters
----------
bias_gid : int
gid of the data to retrieve in the bias data source raster data.
The gids for this data source are the enumerated indices of the
flattened coordinate array.
coord : tuple
(lat, lon) to get data for.
knn : int
Number of nearest neighbors to aggregate from the base data when
comparing to a single site from the bias data.
Expand All @@ -247,6 +266,7 @@ def get_data_pair(self, bias_gid, knn, daily_avg=True):
dist : np.ndarray
Array of nearest neighbor distances with length == knn
"""
bias_gid = self.get_bias_gid(coord)[0]
dist, base_gid = self.get_base_gid(bias_gid, knn)
bias_data = self.get_bias_data(bias_gid)
base_data = self.get_base_data(self.base_fps, self.base_dset, base_gid,
Expand Down Expand Up @@ -349,6 +369,30 @@ class LinearCorrection(DataRetrievalBase):
"""Calculate linear correction *scalar +adder factors to bias correct data
"""

@staticmethod
def get_linear_correction(bias_data, base_data):
"""Get the linear correction factors based on 1D bias and base datasets

Parameters
----------
bias_data : np.ndarray
1D array of biased data observations.
base_data : np.ndarray
1D array of base data observations.

Returns
-------
scalar : float
Factor to adjust the biased data before comparing distributions:
bias_data * scalar + adder
adder : float
Factor to adjust the biased data before comparing distributions:
bias_data * scalar + adder
"""
scalar = (base_data.std() / bias_data.std())
adder = (base_data.mean() - bias_data.mean() * scalar)
return scalar, adder

@classmethod
def _run_single(cls, bias_data, base_fps, base_dset, base_gid,
base_handler, daily_avg):
Expand All @@ -359,13 +403,7 @@ def _run_single(cls, bias_data, base_fps, base_dset, base_gid,
base_gid, base_handler,
daily_avg=daily_avg)

bias_mean = bias_data.mean()
bias_std = bias_data.std()
base_mean = base_data.mean()
base_std = base_data.std()

scalar = (base_std / bias_std)
adder = (base_mean - bias_mean * scalar)
scalar, adder = cls.get_linear_correction(bias_data, base_data)

return scalar, adder

Expand Down