Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

PresRat correction #215

Open
wants to merge 73 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
73 commits
Select commit Hold shift + click to select a range
89fbd29
cfg: Missing to ignore F401
castelao Apr 18, 2024
17f065c
style: Ignore F401 only on __init__
castelao Apr 19, 2024
4d1a49d
Initializing PresRat with reference
castelao Apr 23, 2024
1f490a8
feat: ZeroRateMixin()
castelao Apr 23, 2024
c0f1fcb
refact: Moving ZeroRateMixin to mixins
castelao May 24, 2024
5a8f09a
Adding ipython into dev environment
castelao May 24, 2024
f680a46
fix: ZeroRateMixin shouldn't inherit DataRetrievalBase
castelao May 24, 2024
a31595c
refact: Moving PresRat to qdm sub-module
castelao May 24, 2024
5af4631
fix: Use the given threshold (it was hardcoded before)
castelao May 24, 2024
e9ce57f
PresRat using ZeroRateMixin
castelao May 24, 2024
cbee7c9
doc: Explicit steps for PresRat
castelao May 27, 2024
d6f38e5
Prototype for quantiles on time windows
castelao May 28, 2024
3ceda3f
fix: Missing load PresRat at module
castelao May 28, 2024
c2005cc
Overwritting QDM methods for PresRat requirements
castelao Jun 3, 2024
74facd6
Don't need to overwrite __init__ if defining threshold in the runtime
castelao Jun 3, 2024
f3c4ea9
doc: More details on what is PresRat
castelao Jun 3, 2024
b83108b
Testing apply_zero_precipitation_rate()
castelao Jun 6, 2024
8e2a5c9
test: zero_precipitation_rate()
castelao Jun 11, 2024
a49160a
test: presrat_zero_rate with multiple thresholds
castelao Jun 12, 2024
f66069e
fix, doc: Typo 'monthly'
castelao Jun 12, 2024
4f07699
test: apply_zero_precipitation_rate()
castelao Jun 13, 2024
285015c
test: Minimalist check for test_presrat_transform()
castelao Jun 13, 2024
670ba56
test: parallel results must equal serial
castelao Jun 13, 2024
326133d
style: Fixing code style
castelao Jun 17, 2024
739473b
test: test_presrat_transform_nochanges()
castelao Jun 20, 2024
ecf9f8c
test: Missing to update PresRat standard params for testing
castelao Jun 20, 2024
a2c7a37
test: Adding back a small noise on future projections
castelao Jun 20, 2024
e509792
test: Defining some resources used in the tests
castelao Jun 20, 2024
96eee15
clean, test:
castelao Jun 20, 2024
1c2745b
test: test_presrat_zero_rate with multiple thresholds
castelao Jun 20, 2024
0bb4d60
test, clean: Combining tests
castelao Jun 20, 2024
bdad088
test: A fixture with fut_cc dataset itself for efficiency
castelao Jun 20, 2024
14bd475
test: Standard presrat transform test with dataset from memory
castelao Jun 20, 2024
d0d4fcd
test: test_presrat_transform_nozerochanges()
castelao Jun 20, 2024
d2c8e79
test: test_presrat_transform_nochanges() working
castelao Jun 20, 2024
98360af
style:
castelao Jun 20, 2024
b9272cb
test, doc: test_presrat_transform_nochanges()
castelao Jun 20, 2024
87d5401
Saving used threshold in metadata
castelao Jun 21, 2024
7f8d522
Entry point for presrat functions
castelao Jun 21, 2024
4a7cede
feat: apply_zero_precipitation_rate()
castelao Jun 21, 2024
6af2cf6
feat: get_spatial_bc_presrat()
castelao Jun 21, 2024
4fe27f6
fix, test: test_presrat_transform_nochanges()
castelao Jun 21, 2024
6743a86
feat: Saving requirements to estimate K factor
castelao Jun 21, 2024
b5f21e4
Adding time indices
castelao Jun 21, 2024
811a478
fix: zero_precipitation_rate() dealing with NaN
castelao Jun 21, 2024
df61300
refact: Moved apply_zero_precipitation_rate() into bias_tranforms
castelao Jun 21, 2024
6ef54b8
fix: Missing to load xarray
castelao Jun 21, 2024
c869ab4
refact: get_spatial_bc_quantiles returing params as dict
castelao Jun 24, 2024
0446bb5
feat: local_presrat_bc()
castelao Jun 24, 2024
803a072
style: Mostly moving to single quote which is the standard used in sup3r
castelao Jun 24, 2024
34e4d71
test, clean: PresRat tests were already moved into its own module
castelao Jun 24, 2024
24290c5
Locking numpy to < 2.0
castelao Jun 24, 2024
3d5d5ac
doc: Updating one-line description of PresRat
castelao Jun 24, 2024
7d76c17
test: Updating fut_cc_notrend() to match the 'resource'
castelao Jun 24, 2024
a55262e
test, doc: Better explaing the expected effect of NaNs
castelao Jun 24, 2024
a3f6714
feat: PresRat.write_outputs()
castelao Jun 24, 2024
4bcf101
fix, doc: Weird string literal issue
castelao Jun 24, 2024
9e49792
style: Sorting imports
castelao Jun 24, 2024
112aa00
test, doc: More info in PresRat tests
castelao Jun 24, 2024
39c4f2e
test: Improving test_presrat_calc()
castelao Jun 24, 2024
0b888c4
test, refact: Combining tests
castelao Jun 24, 2024
e08ea76
style: Code formating
castelao Jun 24, 2024
cf2ea97
test, refact: Conforming normalizing dataset
castelao Jun 24, 2024
cefa42e
style: Missing space
castelao Jun 24, 2024
8823368
fix, feat: Missing to include presrat_notrend_params()
castelao Jun 24, 2024
170f722
style: Unused import
castelao Jun 24, 2024
24d1cca
test, fix: Missing a few fixtures
castelao Jun 24, 2024
7b79cf2
doc: Missing documentation on testing functions
castelao Jun 24, 2024
6161227
test: Global zero rate threshold (ZR_THRESHOLD)
castelao Jun 24, 2024
0e94428
clean: Removing ideas for later
castelao Jun 24, 2024
eb7e243
Bump Python to 3.11.9
castelao Jun 25, 2024
c6c788d
style: Conforming with sup3r style
castelao Jun 26, 2024
6a14e66
Adding `workflow_dispatch` as requested by @bnb32
castelao Jun 27, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
test: test_presrat_transform_nochanges() working
  • Loading branch information
castelao committed Jun 20, 2024
commit d2c8e794bd0ae1353486cea38410eb2ed2f7a496
66 changes: 30 additions & 36 deletions tests/bias/test_presrat_bias_correction.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,6 +116,27 @@ def fp_fut_cc_notrend(tmpdir_factory):
return fn


@pytest.fixture(scope='module')
def fut_cc_notrend(fp_fut_cc_notrend):
ds = xr.open_dataset(fp_fut_cc_notrend)
da = ds['rsds'].compute().transpose('lat', 'lon', 'time')
# Unfortunatelly, _get_factors() assume latitude as descending
da = da.sortby('lat', ascending=False)
latlon = np.stack(xr.broadcast(da["lat"], da["lon"] - 360),
axis=-1).astype('float32')
for ii in range(4):
for jj in range(4):
np.allclose(
da.sel(lat=latlon[ii, jj, 0], method='nearest').sel(
lon=latlon[ii, jj, 1] + 360, method='nearest'
)[0],
da.data[ii, jj, 0],
)
assert np.allclose(latlon, FP_CC_LAT_LON)

return da.compute()


@pytest.fixture(scope='module')
def presrat_params(tmpdir_factory, fp_fut_cc):
"""PresRat parameters for standard datasets
Expand Down Expand Up @@ -444,47 +465,20 @@ def test_presrat_transform(presrat_params, fut_cc):
assert not np.allclose(data, corrected, equal_nan=False)


def test_presrat_transform_nochanges(presrat_nochanges_params, fp_fut_cc_notrend):
def test_presrat_transform_nochanges(
presrat_nochanges_params, fut_cc_notrend
):
"""No changes if the three datasets are the same and no zeros"""
# data = np.ones((*FP_CC_LAT_LON.shape[:-1], 3))
# time = pd.to_datetime( ['2015-01-01 12:00:00', '2015-01-02 12:00:00', '2015-01-01 12:00:00'])
breakpoint()
ds = xr.open_dataset(fp_fut_cc_notrend)
da = ds['rsds'].compute().transpose('lat', 'lon', 'time')
# Unfortunatelly, _get_factors() assume latitude as descending
da = da.sortby('lat', ascending=False)
data = da.data
time = pd.to_datetime(da.time)
# latlon = np.stack(xr.broadcast(da["lat"], da["lon"] - 360), axis=-1).astype('float32')
latlon = np.stack(np.meshgrid(da['lon'] - 360, da['lat']), axis=-1)[
:, :, ::-1
].astype('float32')
# da.sel(lat=latlon[ii,jj,0], method="nearest").sel(lon=latlon[ii,jj,1], method="nearest")[0] - data[ii,jj,0]
for ii in range(4):
for jj in range(4):
np.allclose(
da.sel(lat=latlon[ii, jj, 0], method='nearest').sel(
lon=latlon[ii, jj, 1] + 360, method='nearest'
)[0],
data[ii, jj, 0],
)
np.allclose(latlon, FP_CC_LAT_LON)
data = fut_cc_notrend.values
time = pd.to_datetime(fut_cc_notrend.time)
latlon = np.stack(xr.broadcast(fut_cc_notrend["lat"], fut_cc_notrend["lon"] - 360), axis=-1).astype('float32')

# data = np.ones((*FP_CC_LAT_LON.shape[:-1], 3))
# time = pd.to_datetime(
# ['2015-01-01 12:00:00', '2015-01-02 12:00:00', '2015-01-01 12:00:00']
# )
corrected = local_presrat_bc(
data,
time,
latlon,
'ghi',
'rsds',
presrat_nochanges_params,
data, time, latlon, 'ghi', 'rsds', presrat_nochanges_params
)

assert not np.isnan(corrected).all(), "Can't compare if only NaN"
assert np.allclose(data, corrected, equal_nan=False)
assert np.isfinite(corrected).any(), "Can't compare if only NaN"
assert np.allclose(data, corrected, equal_nan=False), "This case shouldn't modify the data"


def test_presrat_transform_nozerochanges(
Expand Down