Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gb/bc #96

Merged
merged 29 commits into from
Oct 4, 2022
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
efcf50e
added bias module with a place for bias transformation functions and …
grantbuster Sep 20, 2022
115e9c2
feature specific bias correction
grantbuster Sep 20, 2022
c184917
removed warnings about excessive padding - not a bad thing
grantbuster Sep 20, 2022
1d823e3
added a site-by-site linear bias correction calculation method
grantbuster Sep 21, 2022
8b7f25b
bug fix and logging
grantbuster Sep 21, 2022
56148db
bias calc mods and new functions
grantbuster Sep 22, 2022
50c3692
added bias calc cli
grantbuster Sep 22, 2022
2f7d4f7
added bias calc to main cli
grantbuster Sep 22, 2022
c132b81
make bias out dir
grantbuster Sep 22, 2022
915edd5
bug fixes and minor refactor to run on eagle
grantbuster Sep 22, 2022
7b28943
added local linear bias correct to forward pass bc options
grantbuster Sep 22, 2022
0cee671
added option to smooth spatial bias correction factors outside of the…
grantbuster Sep 23, 2022
aa0a040
better enumerated progress logging for fwp
grantbuster Sep 23, 2022
240a0d6
added bias correction option to QA
grantbuster Sep 23, 2022
9ac0905
minor refactor to bias correct u and v instead of windspeed and direc…
grantbuster Sep 23, 2022
d2fb1e2
fixed up the u/v QA with bias correction
grantbuster Sep 27, 2022
01fbeda
added meta data to bc h5 output attrs
grantbuster Sep 27, 2022
b370b9f
more bc convenience functions
grantbuster Sep 28, 2022
24c40b6
added monthly bias correction
grantbuster Sep 28, 2022
0deab95
added montly bias correction data transformation method and integrate…
grantbuster Sep 29, 2022
1f28ccc
fixed collection logic for undefined mask meta variable when file is …
grantbuster Oct 2, 2022
5d93283
added bias correction calc tests
grantbuster Oct 3, 2022
0638d7b
added bias transform calcs
grantbuster Oct 3, 2022
6cc6ced
added fwp+bc integration test
grantbuster Oct 3, 2022
ca24793
added qa+bc integration test
grantbuster Oct 3, 2022
b75b3fc
added version record to bias calc output files and incremented versio…
grantbuster Oct 4, 2022
b0a2c49
simplify qa test and pylint issue
grantbuster Oct 4, 2022
7b9c88f
fixed test on h5 meta attrs dtype and docstrings
grantbuster Oct 4, 2022
2ea15e3
serial data handling for QA+BC bug
grantbuster Oct 4, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
added option to smooth spatial bias correction factors outside of the…
… active domain to alleviate the nearest neighbor seams
  • Loading branch information
grantbuster committed Oct 4, 2022
commit 0cee67104d6f15aa25e6828434bef36fb677a102
18 changes: 17 additions & 1 deletion sup3r/bias/bias_calc.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
import pandas as pd
from scipy.spatial import KDTree
from scipy.stats import ks_2samp
from scipy.ndimage.filters import gaussian_filter
from concurrent.futures import ProcessPoolExecutor, as_completed
import rex
from rex.utilities.fun_utils import get_fun_call_str
Expand Down Expand Up @@ -391,7 +392,7 @@ def write_outputs(self, fp_out, scalar, adder):
.format(fp_out))

def run(self, knn, threshold=0.6, fp_out=None, max_workers=None,
daily_avg=True):
daily_avg=True, smoothing=0):
"""Run linear correction factor calculations for every site in the bias
dataset

Expand All @@ -412,6 +413,10 @@ def run(self, knn, threshold=0.6, fp_out=None, max_workers=None,
available.
daily_avg : bool
Flag to do temporal daily averaging of the base data.
smoothing : float
Option to smooth the scalar/adder data outside of the spatial
domain set by the threshold input. This alleviates the weird seams
far from the domain of interest.

Returns
-------
Expand Down Expand Up @@ -473,9 +478,20 @@ def run(self, knn, threshold=0.6, fp_out=None, max_workers=None,
logger.info('Completed bias calculations for {} out of {} '
'sites'.format(i + 1, len(futures)))

logger.info('Finished calculating bias corrections. '
'Mean scalar: {:.3f} mean adder: {:.3f}'
.format(np.nanmean(scalar), np.nanmean(adder)))

nan_mask = np.isnan(scalar)
scalar = nn_fill_array(scalar)
adder = nn_fill_array(adder)

if smoothing > 0:
scalar[nan_mask] = gaussian_filter(scalar, smoothing,
mode='nearest')[nan_mask]
adder[nan_mask] = gaussian_filter(adder, smoothing,
mode='nearest')[nan_mask]

self.write_outputs(fp_out, scalar, adder)

return scalar, adder