diff --git a/docs/src/developers_guide/contributing_running_tests.rst b/docs/src/developers_guide/contributing_running_tests.rst index f60cedba05..a72caa5881 100644 --- a/docs/src/developers_guide/contributing_running_tests.rst +++ b/docs/src/developers_guide/contributing_running_tests.rst @@ -87,7 +87,7 @@ experimental dependency not being present. SKIPPED [1] lib/iris/tests/unit/util/test_demote_dim_coord_to_aux_coord.py:29: Test(s) require external data. All Python decorators that skip tests will be defined in - ``lib/iris/tests/__init__.py`` with a function name with a prefix of + ``lib/iris/tests/_shared_utils.py`` with a function name with a prefix of ``skip_``. You can also run a specific test module. The example below runs the tests for diff --git a/docs/src/developers_guide/contributing_testing.rst b/docs/src/developers_guide/contributing_testing.rst index a65bcebd55..50082d9076 100644 --- a/docs/src/developers_guide/contributing_testing.rst +++ b/docs/src/developers_guide/contributing_testing.rst @@ -45,18 +45,19 @@ When testing a class all the tests must reside in the module: :literal:`lib/iris/tests/unit//test_.py` -Within this test module each tested method must have one or more -corresponding test classes, for example: + +Within this test module each tested method may corresponding test classes, +for example: * ``Test_`` * ``Test___`` -And within those test classes, the test methods must be named according +Within test classes, the test methods must be named according to the aspect of the tested method which they address. **Examples**: -All unit tests for :py:class:`iris.cube.Cube` must reside in: +All unit tests for :py:class:`iris.cube.Cube` reside in: :literal:`lib/iris/tests/unit/cube/test_Cube.py` @@ -95,12 +96,12 @@ When testing a function all the tests must reside in the module: :literal:`lib/iris/tests/unit//test_.py` -Within this test module there must be one or more test classes, for example: +Within this test module there may be test classes, for example: * ``Test`` * ``TestAspectOfFunction`` -And within those test classes, the test methods must be named according +Within those test classes, the test methods must be named according to the aspect of the tested function which they address. **Examples**: diff --git a/docs/src/developers_guide/contributing_testing_index.rst b/docs/src/developers_guide/contributing_testing_index.rst index 2f5ae411e8..faaf8b567a 100644 --- a/docs/src/developers_guide/contributing_testing_index.rst +++ b/docs/src/developers_guide/contributing_testing_index.rst @@ -12,3 +12,5 @@ Testing contributing_running_tests contributing_ci_tests contributing_benchmarks + testing_style_guide + contributing_tests diff --git a/docs/src/developers_guide/contributing_tests.rst b/docs/src/developers_guide/contributing_tests.rst new file mode 100644 index 0000000000..60f0beea6e --- /dev/null +++ b/docs/src/developers_guide/contributing_tests.rst @@ -0,0 +1,301 @@ +.. include:: ../common_links.inc + +.. _contributing_tests: + +*********************** +Contributing Iris Tests +*********************** + +.. _developer_pytest_categories: + +Test Categories +=============== + +There are two main categories of tests within Iris: + +- :ref:`testing.unit_test` +- :ref:`testing.integration` + +Ideally, all code changes should be accompanied by one or more unit +tests, and by zero or more integration tests. + +But if in any doubt about what tests to add or how to write them please +feel free to submit a pull-request in any state and ask for assistance. + + +.. _pytesting.unit_test: + +Unit Tests +---------- + +Code changes should be accompanied by enough unit tests to give a +high degree of confidence that the change works as expected. In +addition, the unit tests can help describe the intent behind a change. + +The docstring for each test module must state the unit under test. +For example: + + :literal:`"""Unit tests for the \`iris.experimental.raster.export_geotiff\` function."""` + +All unit tests must be placed and named according to the following +structure: + + +.. _pytesting.classes: + +Classes +------- + +When testing a class all the tests must reside in the module: + + :literal:`lib/iris/tests/unit//test_.py` + + +Within this test module each tested method may corresponding test classes, +for example: + +* ``Test_`` +* ``Test___`` + +Within test classes, the test methods must be named according +to the aspect of the tested method which they address. + +**Examples**: + +All unit tests for :py:class:`iris.cube.Cube` reside in: + + :literal:`lib/iris/tests/unit/cube/test_Cube.py` + +Within that file the tests might look something like: + +.. code-block:: python + + # Tests for the Cube.xml() method. + class Test_xml(tests.IrisTest): + def test_some_general_stuff(self): + ... + + + # Tests for the Cube.xml() method, focussing on the behaviour of + # the checksums. + class Test_xml__checksum(tests.IrisTest): + def test_checksum_ignores_masked_values(self): + ... + + + # Tests for the Cube.add_dim_coord() method. + class Test_add_dim_coord(tests.IrisTest): + def test_normal_usage(self): + ... + + def test_coord_already_present(self): + ... + + +.. _pytesting.functions: + +Functions +--------- + +When testing a function all the tests must reside in the module: + + :literal:`lib/iris/tests/unit//test_.py` + +Within this test module there may be test classes, for example: + +* ``Test`` +* ``TestAspectOfFunction`` + +Within those test classes, the test methods must be named according +to the aspect of the tested function which they address. + +**Examples**: + +All unit tests for :py:func:`iris.experimental.raster.export_geotiff` +must reside in: + + :literal:`lib/iris/tests/unit/experimental/raster/test_export_geotiff.py` + +Within that file the tests might look something like: + +.. code-block:: python + + # Tests focussing on the handling of different data types. + class TestDtypeAndValues(tests.IrisTest): + def test_int16(self): + ... + + def test_int16_big_endian(self): + ... + + + # Tests focussing on the handling of different projections. + class TestProjection(tests.IrisTest): + def test_no_ellipsoid(self): + ... + + +.. _pytesting.integration: + +Integration Tests +----------------- + +Some code changes may require tests which exercise several units in +order to demonstrate an important consequence of their interaction which +may not be apparent when considering the units in isolation. + +These tests must be placed in the ``lib/iris/tests/integration`` folder. +Unlike unit tests, there is no fixed naming scheme for integration +tests. But folders and files must be created as required to help +developers locate relevant tests. It is recommended they are named +according to the capabilities under test, e.g. +``metadata/test_pp_preservation.py``, and not named according to the +module(s) under test. + +.. _pytesting_style_guide: + +PyTest Style Guide +================== + +This style guide should be approached pragmatically. Most of the guidelines laid out +below will not be practical in every scenario, and as such should not be considered +firm rules. + +At time of writing, some existing tests have already been written in PyTest, +so might not be abiding by these guidelines. + +Directory +--------- + +Where suitable, tests should be located within the relevant directories. +In most circumstance, that means new tests should not be placed in the +root test directory, but in the relevant sub-folders. + +`conftest.py `_ +---------------------------------------------------------------------------------------------------------------------------- + +There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` +folders. Additional lower level conftests can be added if it is agreed there +is a need. + +`Fixtures `_ +------------------------------------------------------------------------------------ + +As far as is possible, the actual test function should do little else but the +actual assertion. Separating off preparation into fixtures may make the code +harder to follow, so compromises are acceptable. For example, setting up a test +``Cube`` should be a fixture, whereas creating a simple string +(``expected = "foo"``), or a single use setup, should *not* be a fixture. + + +New fixtures should always be considered for conftest when added. If it is +decided that they are not suitably reusable, they can be placed within the +local test file. + +`Parameterisation `_ +-------------------------------------------------------------------------------- + +Though it is a useful tool, we should not be complicating tests to work around +parameters; they should only be used when it is simple and apparent to implement. + +Where you are parameterising multiple tests with the same parameters, it is +usually prudent to use the `parameterisation within fixtures +`_. +When doing this, ensure within the tests that it is apparent that they are being +parameterised, either within the fixture name or with comments. + +All parameterisation benefits from +`ids `_, +and so should be used where possible. + +`Classes `_ +--------------------------------------------------------------------------------------------------- + +How and when to group tests within classes can be based on personal opinion, +we do not deem consistency on this a vital concern. + +`Mocks `_ +-------------------------------------------------------------------- + +Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable. + +.. note:: + If you think we're missing anything important here, please consider creating an + issue or discussion and share your ideas with the team! + +.. _pytesting_tools: + +============= +Testing tools +============= + +.. note:: + :class:`iris.tests.IrisTest` has been deprecated, and replaced with + the :mod:`iris.tests._shared_utils` module. + +Iris has various internal convenience functions and utilities available to +support writing tests. Using these makes tests quicker and easier to write, and +also consistent with the rest of Iris (which makes it easier to work with the +code). Most of these conveniences are accessed through the +:mod:`iris.tests._shared_utils` module. + +.. tip:: + + All functions listed on this page are defined within + :mod:`iris.tests._shared_utils`. They can be accessed within a test using + ``_shared_utils.exampleFunction``. + +Custom assertions +----------------- + +:mod:`iris.tests._shared_utils` supports a variety of custom pytest-style +assertions, such as :meth:`~iris.tests._shared_utils.assert_array_equal`, and +:meth:`~iris.tests._shared_utils.assert_array_almost_equal`. + +.. _pycreate-missing: + +Saving results +-------------- + +Some tests compare the generated output to the expected result contained in a +file. Custom assertions for this include +:meth:`~iris.tests._shared_utils.assert_CML_approx_data` +:meth:`~iris.tests._shared_utils.assert_CDL` +:meth:`~iris.tests._shared_utils.assert_CML` and +:meth:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more +information. + +.. note:: + + Sometimes code changes alter the results expected from a test containing the + above methods. These can be updated by removing the existing result files + and then running the file containing the test with a ``--create-missing`` + command line argument, or setting the ``IRIS_TEST_CREATE_MISSING`` + environment variable to anything non-zero. This will create the files rather + than erroring, allowing you to commit the updated results. + +Context managers +---------------- + +Capturing exceptions and logging +-------------------------------- + +:mod:`~iris.tests._shared_utils` includes several context managers that can be used +to make test code tidier and easier to read. These include +:meth:`~iris.tests.IrisTest_nometa.assert_no_warnings_regexp` and +:meth:`~iris.tests.IrisTest_nometa.assert_logs`. + +Patching +-------- + +After the change from ``unittest`` to ``pytest``, ``IrisTest.patch`` has been +converted into :meth:`~iris.tests._shared_utils.patch`. + +This is currently not implemented, and will raise an error if called. + +Graphic tests +------------- + +As a package capable of generating graphical outputs, Iris has utilities for +creating and updating graphical tests - see :ref:`testing.graphics` for more +information. \ No newline at end of file diff --git a/docs/src/developers_guide/testing_style_guide.rst b/docs/src/developers_guide/testing_style_guide.rst new file mode 100644 index 0000000000..8d8520e077 --- /dev/null +++ b/docs/src/developers_guide/testing_style_guide.rst @@ -0,0 +1,73 @@ +.. include:: ../common_links.inc + +.. _testing_style_guide: + +PyTest Style Guide +****************** + +This style guide should be approached pragmatically. Most of the guidelines laid out +below will not be practical in every scenario, and as such should not be considered +firm rules. + +At time of writing, some existing tests have already been written in PyTest, +so might not be abiding by these guidelines. + +Directory +========= + +Where suitable, tests should be located within the relevant directories. +In most circumstance, that means new tests should not be placed in the +root test directory, but in the relevant sub-folders. + +`conftest.py `_ +============================================================================================================================ + +There should be a ``conftest.py`` file in the ``root/unit`` and ``root/integration`` +folders. Additional lower level conftests can be added if it is agreed there +is a need. + +`Fixtures `_ +==================================================================================== + +As far as is possible, the actual test function should do little else but the +actual assertion. Separating off preparation into fixtures may make the code +harder to follow, so compromises are acceptable. For example, setting up a test +``Cube`` should be a fixture, whereas creating a simple string +(``expected = "foo"``), or a single use setup, should *not* be a fixture. + + +New fixtures should always be considered for conftest when added. If it is +decided that they are not suitably reusable, they can be placed within the +local test file. + +`Parameterisation `_ +================================================================================ + +Though it is a useful tool, we should not be complicating tests to work around +parameters; they should only be used when it is simple and apparent to implement. + +Where you are parameterising multiple tests with the same parameters, it is +usually prudent to use the `parameterisation within fixtures +`_. +When doing this, ensure within the tests that it is apparent that they are being +parameterised, either within the fixture name or with comments. + +All parameterisation benefits from +`ids `_, +and so should be used where possible. + +`Classes `_ +=================================================================================================== + +How and when to group tests within classes can be based on personal opinion, +we do not deem consistency on this a vital concern. + +`Mocks `_ +==================================================================== + +Any mocking should be done with ``pytest.mock``, and monkeypatching where suitable. + +.. note:: + If you think we're missing anything important here, please consider creating an + issue or discussion and share your ideas with the team! + diff --git a/docs/src/developers_guide/testing_tools.rst b/docs/src/developers_guide/testing_tools.rst index dd628d37fc..ea9343ee7e 100755 --- a/docs/src/developers_guide/testing_tools.rst +++ b/docs/src/developers_guide/testing_tools.rst @@ -5,27 +5,28 @@ Testing tools ************* +.. note:: + :class:`iris.tests.IrisTest` has been deprecated, and replaced with + the :mod:`iris.tests._shared_utils` module. + Iris has various internal convenience functions and utilities available to support writing tests. Using these makes tests quicker and easier to write, and also consistent with the rest of Iris (which makes it easier to work with the code). Most of these conveniences are accessed through the -:class:`iris.tests.IrisTest` class, from -which Iris' test classes then inherit. +:mod:`iris.tests._shared_utils` module. .. tip:: All functions listed on this page are defined within - :mod:`iris.tests.__init__.py` as methods of - :class:`iris.tests.IrisTest_nometa` (which :class:`iris.tests.IrisTest` - inherits from). They can be accessed within a test using - ``self.exampleFunction``. + :mod:`iris.tests._shared_utils`. They can be accessed within a test using + ``_shared_utils.exampleFunction``. Custom assertions ================= -:class:`iris.tests.IrisTest` supports a variety of custom unittest-style -assertions, such as :meth:`~iris.tests.IrisTest_nometa.assertArrayEqual`, -:meth:`~iris.tests.IrisTest_nometa.assertArrayAlmostEqual`. +:mod:`iris.tests._shared_utils` supports a variety of custom pytest-style +assertions, such as :meth:`~iris.tests._shared_utils.assert_array_equal`, and +:meth:`~iris.tests._shared_utils.assert_array_almost_equal`. .. _create-missing: @@ -34,10 +35,10 @@ Saving results Some tests compare the generated output to the expected result contained in a file. Custom assertions for this include -:meth:`~iris.tests.IrisTest_nometa.assertCMLApproxData` -:meth:`~iris.tests.IrisTest_nometa.assertCDL` -:meth:`~iris.tests.IrisTest_nometa.assertCML` and -:meth:`~iris.tests.IrisTest_nometa.assertTextFile`. See docstrings for more +:meth:`~iris.tests._shared_utils.assert_CML_approx_data` +:meth:`~iris.tests._shared_utils.assert_CDL` +:meth:`~iris.tests._shared_utils.assert_CML` and +:meth:`~iris.tests._shared_utils.assert_text_file`. See docstrings for more information. .. note:: @@ -55,22 +56,18 @@ Context managers Capturing exceptions and logging -------------------------------- -:class:`iris.tests.IrisTest` includes several context managers that can be used +:mod:`~iris.tests._shared_utils` includes several context managers that can be used to make test code tidier and easier to read. These include -:meth:`~iris.tests.IrisTest_nometa.assertWarnsRegexp` and -:meth:`~iris.tests.IrisTest_nometa.assertLogs`. - -Temporary files ---------------- - -It's also possible to generate temporary files in a concise fashion with -:meth:`~iris.tests.IrisTest_nometa.temp_filename`. +:meth:`~iris.tests.IrisTest_nometa.assert_no_warnings_regexp` and +:meth:`~iris.tests.IrisTest_nometa.assert_logs`. Patching ======== -:meth:`~iris.tests.IrisTest_nometa.patch` is a wrapper around ``unittest.patch`` -that will be automatically cleaned up at the end of the test. +After the change from ``unittest`` to ``pytest``, ``IrisTest.patch`` has been +converted into :meth:`~iris.tests._shared_utils.patch`. + +This is currently not implemented, and will raise an error if called. Graphic tests =============