Fill out pytest suite #2549

loriab · 2022-04-19T17:40:07Z

Description

A few testing bugs and completing the pytest access route. Nothing controversial. Most of this PR is autogenerated. Due diligence on review can be achieved by examining:

empirical_dispersion.py
addons.py
extern1/input.dat
tester.py
test_addons.py

Todos

fixed bug that wouldn't allow plain pytest of installed psi4/ b/c would bring in incorrectly configured qcdb tests. workaround was to pytest psi4/tests/
add a couple corrections I had promised to forward-port ddd convergence and derivative logic #2536 but had forgotten to push
fixes Single cpu job fails in qcengine/config.py with KeyError #2548 by passing psi4's -n to qcng for dftd3/gcp/mp2d
corrects dfmp2-freq2 ctest was registered as dfmp2-freq1
I realized externalpotentials weren't getting unset after a energy/grad/etc. command. This is probably a new bug after ExternalPotential in Bohr #2515 but didn't confirm. It's healed in DDD, but the solution didn't port, so I added the test and temporary workaround to extern1
Added feature to ctest_runner to allow directory structure to be copied into test scratch for psithon2. This is proven to work but needs a couple lines added to qcengine, so hidden for now.
Fixed resp addon tests that always failed in parallel. these write to fixed-name files that aren't configurable, so they need to be chdir'd, if not run serially.
Added a script tester.py that checks some config stuff and writes out missing test_input.py. See the goals there for details. Running it produces a list of flaws like the below. I'll post this to an issue for later attention. Someday this can be hooked up to GHA and replace the perl script.

Complaints
----------
- [ ]   1. cc5: missing cmake directory registration. `vi CMakeLists.txt`
- [ ]   2. cookbook/manual-sow-reap: missing cmake directory registration. `vi cookbook/CMakeLists.txt`
- [ ]   3. cookbook/manual-sow-reap: missing CMakeLists. `vi cookbook/manual-sow-reap/CMakeLists.txt`
- [ ]   4. dfmp2-freq1: missing cmake directory registration. `vi CMakeLists.txt`
- [ ]   5. dfmp2-freq2: missing cmake directory registration. `vi CMakeLists.txt`
- [ ]   6. dfomp2p5-1: missing ctest registration. `vi dfomp2p5-1/CMakeLists.txt` ...
- [ ]  73. v2rdm_casscf/v2rdm7: mismatched marks ctest (opt;v2rdm) and pytest (opt). `vi v2rdm_casscf/v2rdm7/CMakeLists.txt v2rdm_casscf/v2rdm7/test_input.py`
- [ ]  74. x2c-perturb-h: mismatched directory (x2c-perturb-h) and ctest registration name (x2c-perturb_h). `vi x2c-perturb-h/CMakeLists.txt`

Filled in the rest of the test_input's as generated from script. There's ~3 hidden, but otherwise, the full test suite can be run from pytest. On 20 cores, I get 0 failed, 3660 passed, 96 skipped, 272 xfailed, 4 xpassed, 312 warnings in 2011.32s (0:33:31). For interest, the culprits over 2 minutes are:

=========================================================================================================== slowest 50 durations ============================================================================================================
937.48s call     tests/linK-2/test_input.py::test_linK_2
875.59s call     tests/snsmp2/cc-cc/test_input.py::test_snsmp2_cc_cc
855.00s call     tests/fd-freq-energy-large/test_input.py::test_fd_freq_energy_large
569.12s call     tests/density-screen-2/test_input.py::test_density_screen_2
499.66s call     tests/fsapt1/test_input.py::test_fsapt1
242.29s call     tests/snsmp2/cf-o/test_input.py::test_snsmp2_cf_o
229.90s call     tests/opt13/test_input.py::test_opt13
221.20s call     tests/sapt-exch-ind30-inf/test_input.py::test_sapt_exch_ind30_inf
204.84s call     tests/chemps2/caspt2-n2/test_input.py::test_chemps2_caspt2_n2
197.04s call     tests/test_dft_benchmarks.py::test_dft_bench_interaction[wB97M-V- - ]
193.31s call     tests/cbs-xtpl-func/test_input.py::test_cbs_xtpl_func
189.76s call     tests/test_dft_benchmarks.py::test_dft_bench_interaction[LC-VV10- - ]
183.86s call     tests/isapt1/test_input.py::test_isapt1
181.63s call     tests/test_dft_benchmarks.py::test_dft_bench_interaction[wB97X-V- - ]
174.31s call     tests/test_dft_benchmarks.py::test_dft_bench_interaction[B97M-V- - ]
171.48s call     tests/sapt4/test_input.py::test_sapt4
169.47s call     tests/test_dft_benchmarks.py::test_dft_bench_interaction[VV10- - ]
160.20s call     tests/fd-freq-gradient-large/test_input.py::test_fd_freq_gradient_large
157.12s call     tests/nbody-hessian/test_input.py::test_nbody_hessian
135.53s call     tests/cc13a/test_input.py::test_cc13a

ADDED: remove the remains of sowreap testing closes Remove sowreap from runtest.py #2554
In future PR, the CI should be adjusted so CTest is running, say, smoke tests, and pytest is running quick. Otherwise there'll be double testing.

Checklist

Tests added for any new features
All or relevant fraction of full tests run

Status

Ready for review
Ready for merge

JonathonMisiewicz

I'll be in touch when it's time to kill autotest. We're close.

pytest.ini

zachglick

I see that the tests in tests/optking are nested and didn't get individual python files. Is this intentional / okay? Otherwise LGTM!

zachglick · 2022-05-04T19:41:52Z

tests/runtest.py

@@ -32,34 +32,33 @@
 import time
 import subprocess

-if len(sys.argv) not in {4, 5, 6, 7, 8, 9}:
- print("""Usage: %s input_file logfile top_srcdir doreaptest alt_output_file alt_psi4_exe alt_psi4datadir""" % (sys.argv[0]))
+if len(sys.argv) not in {4, 5, 6, 7, 8}:


It might be good to eventually argparse this if the arguments here will change in the future.

Agree that argparse is the sensible way. But this is called with fixed argument list from CMake, so it doesn't really need to be helpful or flexible. Another possibility is to replace the whole fn by a call to qcengine.execute.

loriab · 2022-05-04T19:58:29Z

I see that the tests in tests/optking are nested and didn't get individual python files. Is this intentional / okay? Otherwise LGTM!

Yes, I deferred all the cleanup to #2555, and RAK is fine with deleting most.

loriab added the testing label Apr 19, 2022

loriab added this to the Psi4 1.6 milestone Apr 19, 2022

JonathonMisiewicz approved these changes Apr 19, 2022

View reviewed changes

loriab added 4 commits April 22, 2022 10:33

touchups to psi4#2536

e9eecc1

pytest all quick

37cc469

fill out pytest for psithon test suite

afcc48b

stragglers

2ccf9e6

JonathonMisiewicz reviewed Apr 22, 2022

View reviewed changes

pytest.ini Outdated Show resolved Hide resolved

remove sowreap testing

0ca1024

loriab force-pushed the quicktests branch from 8c042f7 to 0ca1024 Compare April 22, 2022 16:01

loriab mentioned this pull request Apr 22, 2022

clean up tests directory #2555

Closed

75 tasks

maxscheurer approved these changes Apr 27, 2022

View reviewed changes

zachglick approved these changes May 4, 2022

View reviewed changes

loriab merged commit f8a5f63 into psi4:master May 4, 2022

loriab deleted the quicktests branch May 4, 2022 19:59

loriab mentioned this pull request May 11, 2022

Bring PSIthon tests to pytest's attention #2495

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fill out pytest suite #2549

Fill out pytest suite #2549

loriab commented Apr 19, 2022 •

edited

Loading

JonathonMisiewicz left a comment

zachglick left a comment

zachglick May 4, 2022

loriab May 4, 2022

loriab commented May 4, 2022

Fill out pytest suite #2549

Fill out pytest suite #2549

Conversation

loriab commented Apr 19, 2022 • edited Loading

Description

Todos

Checklist

Status

JonathonMisiewicz left a comment

Choose a reason for hiding this comment

zachglick left a comment

Choose a reason for hiding this comment

zachglick May 4, 2022

Choose a reason for hiding this comment

loriab May 4, 2022

Choose a reason for hiding this comment

loriab commented May 4, 2022

loriab commented Apr 19, 2022 •

edited

Loading