Improved QA4SM metrics calculators (#219)

* temporal matching for multiple frames

* don't convert timezone naive indices

* combined matcher for dictionary

* added option to use all with combined_dropna

* comparison of new and old matching implementation

This adds a test to check whether the new matching in
BasicTemporalMatching works as the old one in case there are no
duplicates in the reference frame.

It replaced the matcher in the other tests by a dummy that still uses
the old matching implementation.
The following commit will remove the dummy and set the new expected
values.

* changed tests for ascat_ismn to use new matching

* fixup for previous commit

* added PairwiseIntercomparisonMetrics

* fixed test for PairwiseIntercomparisonMetrics

* statistics based metrics calculator

* Added tests for PairwiseIntercomparisonMetrics

One test still fails due to difference in scaling before calculating
ubRMSD.

* Added CIs to PairwiseIntercomparisonMetrics

* made bootstrapped CI tests less sensitive

* made bootstrapped CI tests less sensitive

* added missing import

* temporal matching for multiple frames

* don't convert timezone naive indices

* added option to use all with combined_dropna

* combined matcher for dictionary

* comparison of new and old matching implementation

This adds a test to check whether the new matching in
BasicTemporalMatching works as the old one in case there are no
duplicates in the reference frame.

It replaced the matcher in the other tests by a dummy that still uses
the old matching implementation.
The following commit will remove the dummy and set the new expected
values.

* changed tests for ascat_ismn to use new matching

* fixup for previous commit

* added PairwiseIntercomparisonMetrics

* fixed test for PairwiseIntercomparisonMetrics

* statistics based metrics calculator

* Added tests for PairwiseIntercomparisonMetrics

One test still fails due to difference in scaling before calculating
ubRMSD.

* Added CIs to PairwiseIntercomparisonMetrics

* made bootstrapped CI tests less sensitive

* made bootstrapped CI tests less sensitive

* added missing import

* make_combined_temporal_matcher now also works with k>2

* Fixed some issues

- bug in validation.get_data_for_result_tuple
- no segfault when running tests anymore

* better confidence intervals for tcol

* added tests for tripletmetrics

* added imports

* run tests all at the same time

* removed imports from validation_framework init

* triple metrics only

* removed xarray test

* fixed test

* fixed test

* TripleIntercomparisonMetrics -> TripleCollocationMetrics

* use percentile method by default; analytical CIs for mse_bias

* better test for #220

.github/workflows/ci.yml

@@ -54,8 +54,7 @@ jobs:
  shell: bash -l {0}
  run: |
  pip install .
- pytest --cache-clear -m 'not full_framework'
- pytest --cache-clear -m 'full_framework'
+ pytest --cache-clear
  - name: Upload Coverage
  shell: bash -l {0}
  run: |

.pre-commit-config.yaml

@@ -7,7 +7,6 @@ repos:
  - id: trailing-whitespace
  - id: end-of-file-fixer
  - id: check-yaml
- - id: check-added-large-files
 - repo: https://gitlab.com/pycqa/flake8
  rev: 3.7.9
  hooks:

DEVELOPERS_GUIDE.md

@@ -31,5 +31,7 @@ after you applied your changes. There will be some warnings like this:
 
  warning: src/pytesmo/metrics/_fast.pyx:11:6: Unused entry 'dtype_signed'
 
-Ignore the warnings about 'dtype_signed'. If there are other warnings, you
-should probably investigate them.
+Ignore the warnings about unused entries, e.g. 'dtype_signed', 'itemsize',
+'kind', 'memslice', 'ndarray', and maybe some more. If there are other warnings
+than unused entry, or if one of the unused entries looks like a variable name
+you used, you should probably investigate them.

setup.cfg

@@ -107,6 +107,11 @@ filterwarnings =
  ignore:An input array is constant:
  # some of the scaling tests cause this
  ignore:The bins have been resized:UserWarning
+ # this is due to the test data
+ ignore:Dropping duplicated indices in reference:UserWarning
+ ignore:No timezone given for reference:UserWarning
+ # this comes from somewhere else
+ ignore:`np.bool` is a deprecated alias for the builtin `bool`
 
 [aliases]
 dists = bdist_wheel
@@ -128,6 +133,7 @@ formats = bdist_wheel
 [flake8]
 # Some sane defaults for the code style checker flake8
 extend-ignore = F403
+per-file-ignores = __init__.py:F401
 exclude =
  .tox
  build

setup.py

@@ -1,14 +1,4 @@
 # -*- coding: utf-8 -*-
-"""
- Setup file for pytesmo.
- Use setup.cfg to configure your project.
-
- This file was generated with PyScaffold 3.2.3.
- PyScaffold helps you to put up the scaffold of your new Python project.
- Learn more under: https://pyscaffold.org/
-
- The code to cythonize extensions was added manually.
-"""
 
 from setuptools import setup
 from setuptools.command.build_ext import build_ext as _build_ext
@@ -32,13 +22,20 @@ def get_ext_modules(ext):
  include_dirs=[numpy.get_include()],
  define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
  ),
+ Extension(
+ "pytesmo.metrics._fast_pairwise",
+ ["src/pytesmo/metrics/_fast_pairwise" + ext],
+ include_dirs=[numpy.get_include()],
+ define_macros=[("NPY_NO_DEPRECATED_API", "NPY_1_7_API_VERSION")],
+ ),
  ]
 
 
 # defining a custom cythonize function that sets all the options we want and
 # that can be reused for the different options
 def cythonize_extensions():
  from Cython.Build import cythonize
+
  cythonize(
  get_ext_modules(".pyx"),
  compiler_directives={
@@ -68,10 +65,9 @@ def run(self):
 
 class build_ext(_build_ext):
 
- user_options = (
- getattr(_build_ext, "user_options", [])
- + [("cythonize", None, "recreate the C extensions with cython")]
- )
+ user_options = getattr(_build_ext, "user_options", []) + [
+ ("cythonize", None, "recreate the C extensions with cython")
+ ]
 
  def initialize_options(self):
  super().initialize_options()
@@ -83,12 +79,12 @@ def run(self):
  super().run()
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
  cmdclass = {}
  cmdclass["sdist"] = sdist
  cmdclass["build_ext"] = build_ext
  setup(
  cmdclass=cmdclass,
  # at this point the C modules have already been generated if necessary
- ext_modules=get_ext_modules(".c")
+ ext_modules=get_ext_modules(".c"),
  )

src/pytesmo/__init__.py

@@ -32,4 +32,5 @@
  "time_series",
  "grid",
  "colormaps",
+ "validation_framework"
 ]

src/pytesmo/metrics/__init__.py

@@ -28,8 +28,7 @@
 * rmsd
 * nrmsd
 * ubrmsd
-* mean_squared_error
-* mse_bias
+* mse
 * pearson_r
 * spearman_r
 * kendall_tau
@@ -68,35 +67,33 @@
 
 from pytesmo.metrics.pairwise import (
  aad,
+ bias,
  mad,
  msd,
  rmsd,
  nrmsd,
  ubrmsd,
- pearson_r,
- spearman_r,
- kendall_tau,
- nash_sutcliffe,
- index_of_agreement,
-)
-from pytesmo.metrics._fast import (
- bias,
  mse_bias,
  mse_var,
  mse_corr,
  mse_decomposition,
  RSS,
+ pearson_r,
+ spearman_r,
+ kendall_tau,
+ nash_sutcliffe,
+ index_of_agreement,
  rolling_pr_rmsd,
 )
-from pytesmo.metrics.pairwise_utils import (
+from pytesmo.metrics.confidence_intervals import (
  has_analytical_ci,
  with_analytical_ci,
  with_bootstrapped_ci,
+ tcol_metrics_with_bootstrapped_ci,
 )
 
 from pytesmo.metrics.tcol import (
  tcol_metrics,
- tcol_metrics_with_bootstrapped_ci,
  ecol
 )