dftd3 only works on Unix systems

[sjayellis]

Pytests using dftd3 fail on non-Unix systems.

[loriab]

you'll need to be more specific for help. is that windows or mac that's failing? what is the error? what is conda list?

[sjayellis]

I am working with the Ubuntu WSL on a Windows machine. The following two tests are failing:

qcengine/tests/test_standard_suite.py::test_compute_energy[dftd3-model0] FAILED
qcengine/tests/test_standard_suite.py::test_compute_gradient[dftd3-model0] FAILED

Running each test individually provides the following output:

>>> pytest qcengine/tests/test_standard_suite.py::test_compute_energy
===================================== test session starts ======================================
platform linux -- Python 3.7.1, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /c/Users/samel/Desktop/Molssi/QCArchive/QCEngine, inifile: setup.cfg
plugins: cov-2.6.1
collected 4 items

qcengine/tests/test_standard_suite.py F.ss                                               [100%]

=========================================== FAILURES ===========================================
______________________________ test_compute_energy[dftd3-model0] _______________________________

program = 'dftd3', model = {'method': 'b3lyp-d3'}

    @pytest.mark.parametrize("program, model", _canonical_methods)
    def test_compute_energy(program, model):
        if not testing.has_program(program):
            pytest.skip("Program '{}' not found.".format(program))


        inp = ResultInput(molecule=qcng.get_molecule("hydrogen"), driver="energy", model=model)
>       ret = qcng.compute(inp, program, raise_error=True)

qcengine/tests/test_standard_suite.py:27:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
qcengine/compute.py:103: in compute
    return handle_output_metadata(output_data, metadata, raise_error=raise_error, return_dict=return_dict)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

output_data = <ResultInput molecule=<    Geometry (in Angstrom), charge = 0.0, multiplicity = 1:

       Center    … driver=<DriverE...on=1 keywords={} extras={} provenance={'creator': 'QCElemental', 'version': 'v0.3.1', 'routine': 'qcelemental.models.…>
metadata = {'error_message': 'QCEngine Call Error:\nTraceback (most recent call last):\n  File "/c/Users/samel/Desktop/Molssi/QCA...ful run. Possibly -D variant not available in dftd3 version.\n', 'stderr': None, 'stdout': None, 'success': False, ...}
raise_error = True, return_dict = False

    def handle_output_metadata(output_data: Union[Dict[str, Any], 'BaseModel'],
                               metadata: Dict[str, Any],
                               raise_error: bool = False,
                               return_dict: bool = True) -> Union[Dict[str, Any], 'BaseModel']:
        """
        Fuses general metadata and output together.

        Returns
        -------
        result : dict or pydantic.models.Result
            Output type depends on return_dict or a dict if an error was generated in model construction
        """

        if isinstance(output_data, dict):
            output_fusion = output_data  # Error handling
        else:
            output_fusion = output_data.dict()

        # Do not override if computer generates
        output_fusion["stdout"] = output_fusion.get("stdout", None) or metadata["stdout"]
        output_fusion["stderr"] = output_fusion.get("stderr", None) or metadata["stderr"]

        if metadata["success"] is not True:
            output_fusion["success"] = False
            output_fusion["error"] = {"error_type": "meta_error", "error_message": metadata["error_message"]}

        # Raise an error if one exists and a user requested a raise
        if raise_error and (output_fusion["success"] is not True):
            msg = "stdout:\n{}".format(output_fusion["stdout"])
            msg += "\nstderr:\n{}".format(output_fusion["stderr"])
            LOGGER.info(msg)
>           raise ValueError(output_fusion["error"]["error_message"])
E           ValueError: QCEngine Call Error:
E           Traceback (most recent call last):
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/util.py", line 77, in compute_wrapper
E               yield metadata
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/compute.py", line 101, in compute
E               output_data = executor.compute(input_data, config)
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 55, in compute
E               output_data = run_json(input_data)
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 111, in run_json
E               raise exc
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 103, in run_json
E               dftd3_driver(jobrec)
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 248, in dftd3_driver
E               jobrec=jobrec, module_label='dftd3', plant=dftd3_plant, harvest=dftd3_harvest, verbose=verbose)
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 305, in module_driver
E               harvest(jobrec, modulerec)  # updates jobrec
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 456, in dftd3_harvest
E               raise ValueError('Unsuccessful run. Possibly -D variant not available in dftd3 version.')
E           ValueError: Unsuccessful run. Possibly -D variant not available in dftd3 version.

qcengine/util.py:155: ValueError
======================== 1 failed, 1 passed, 2 skipped in 2.00 seconds =========================

>>> pytest qcengine/tests/test_standard_suite.py::test_compute_gradient
===================================== test session starts ======================================
platform linux -- Python 3.7.1, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /c/Users/samel/Desktop/Molssi/QCArchive/QCEngine, inifile: setup.cfg
plugins: cov-2.6.1
collected 4 items

qcengine/tests/test_standard_suite.py F.ss                                               [100%]

=========================================== FAILURES ===========================================
_____________________________ test_compute_gradient[dftd3-model0] ______________________________

program = 'dftd3', model = {'method': 'b3lyp-d3'}

    @pytest.mark.parametrize("program, model", _canonical_methods)
    def test_compute_gradient(program, model):
        if not testing.has_program(program):
            pytest.skip("Program '{}' not found.".format(program))

        inp = ResultInput(molecule=qcng.get_molecule("hydrogen"), driver="gradient", model=model)
>       ret = qcng.compute(inp, program, raise_error=True)

qcengine/tests/test_standard_suite.py:39:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
qcengine/compute.py:103: in compute
    return handle_output_metadata(output_data, metadata, raise_error=raise_error, return_dict=return_dict)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

output_data = <ResultInput molecule=<    Geometry (in Angstrom), charge = 0.0, multiplicity = 1:

       Center    … driver=<DriverE...on=1 keywords={} extras={} provenance={'creator': 'QCElemental', 'version': 'v0.3.1', 'routine': 'qcelemental.models.…>
metadata = {'error_message': 'QCEngine Call Error:\nTraceback (most recent call last):\n  File "/c/Users/samel/Desktop/Molssi/QCA...ful run. Possibly -D variant not available in dftd3 version.\n', 'stderr': None, 'stdout': None, 'success': False, ...}
raise_error = True, return_dict = False

    def handle_output_metadata(output_data: Union[Dict[str, Any], 'BaseModel'],
                               metadata: Dict[str, Any],
                               raise_error: bool = False,
                               return_dict: bool = True) -> Union[Dict[str, Any], 'BaseModel']:
        """
        Fuses general metadata and output together.

        Returns
        -------
        result : dict or pydantic.models.Result
            Output type depends on return_dict or a dict if an error was generated in model construction
        """

        if isinstance(output_data, dict):
            output_fusion = output_data  # Error handling
        else:
            output_fusion = output_data.dict()

        # Do not override if computer generates
        output_fusion["stdout"] = output_fusion.get("stdout", None) or metadata["stdout"]
        output_fusion["stderr"] = output_fusion.get("stderr", None) or metadata["stderr"]

        if metadata["success"] is not True:
            output_fusion["success"] = False
            output_fusion["error"] = {"error_type": "meta_error", "error_message": metadata["error_message"]}

        # Raise an error if one exists and a user requested a raise
        if raise_error and (output_fusion["success"] is not True):
            msg = "stdout:\n{}".format(output_fusion["stdout"])
            msg += "\nstderr:\n{}".format(output_fusion["stderr"])
            LOGGER.info(msg)
>           raise ValueError(output_fusion["error"]["error_message"])
E           ValueError: QCEngine Call Error:
E           Traceback (most recent call last):
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/util.py", line 77, in compute_wrapper
E               yield metadata
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/compute.py", line 101, in compute
E               output_data = executor.compute(input_data, config)
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 55, in compute
E               output_data = run_json(input_data)
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 111, in run_json
E               raise exc
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 103, in run_json
E               dftd3_driver(jobrec)
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 248, in dftd3_driver
E               jobrec=jobrec, module_label='dftd3', plant=dftd3_plant, harvest=dftd3_harvest, verbose=verbose)
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 305, in module_driver
E               harvest(jobrec, modulerec)  # updates jobrec
E             File "/c/Users/samel/Desktop/Molssi/QCArchive/QCEngine/qcengine/programs/dftd3/runner.py", line 456, in dftd3_harvest
E               raise ValueError('Unsuccessful run. Possibly -D variant not available in dftd3 version.')
E           ValueError: Unsuccessful run. Possibly -D variant not available in dftd3 version.

qcengine/util.py:155: ValueError
======================== 1 failed, 1 passed, 2 skipped in 2.11 seconds =========================

Here is the conda list output. My environment was generated using:
python devtools/scripts/conda_env.py -n=qcengine -p=3.7 devtools/conda-envs/psi.yaml
within the root directory.

>>> conda list
# packages in environment at /home/sjellis/anaconda3/envs/qcengine:
#
# Name                    Version                   Build  Channel
alabaster                 0.7.12                   py37_0
ambit                     0.2                  he865b90_0    psi4
asn1crypto                0.24.0                   py37_0
atomicwrites              1.3.0                      py_0    conda-forge
attrs                     19.1.0                     py_0    conda-forge
babel                     2.6.0                    py37_0
blas                      1.0                         mkl
bzip2                     1.0.6             h14c3975_1002    conda-forge
ca-certificates           2019.1.23                     0
certifi                   2018.11.29               py37_0
cffi                      1.12.1           py37h2e261b9_0
chardet                   3.0.4                    pypi_0    pypi
chemps2                   1.8.9                h8c3debe_0    psi4
codecov                   2.0.15                   pypi_0    pypi
coverage                  4.5.2           py37h14c3975_1001    conda-forge
cryptography              2.5              py37h1ba5d50_0
decorator                 4.3.2                      py_0    conda-forge
deepdiff                  3.3.0                    py37_1    psi4
dftd3                     3.2.1                h0e1e685_0    psi4
dkh                       1.2                  h173d85e_2    psi4
docutils                  0.14                     py37_0
gau2grid                  1.3.1                h035aef0_0    psi4
gdma                      2.2.6                h0e1e685_6    psi4
geometric                 0.9.4+2.g08fd110          pypi_0    pypi
hdf5                      1.10.2               hc401514_3    conda-forge
idna                      2.8                      pypi_0    pypi
imagesize                 1.1.0                    py37_0
intel-openmp              2019.1                      144
jinja2                    2.10                     py37_0
jsonpickle                0.9.6                    py37_0    psi4
libffi                    3.2.1             hf484d3e_1005    conda-forge
libgcc-ng                 7.3.0                hdf63c60_0    conda-forge
libgfortran               3.0.0                         1    conda-forge
libgfortran-ng            7.2.0                hdf63c60_3    conda-forge
libint                    1.2.1                hb4a4fd4_6    psi4
libstdcxx-ng              7.3.0                hdf63c60_0    conda-forge
libxc                     4.3.3                h7b6447c_0    psi4
markupsafe                1.1.1            py37h7b6447c_0
mkl                       2019.1                      144
mkl_fft                   1.0.10           py37h14c3975_1    conda-forge
mkl_random                1.0.2            py37h637b7d7_2    conda-forge
more-itertools            4.3.0                 py37_1000    conda-forge
ncurses                   6.1               hf484d3e_1002    conda-forge
networkx                  2.2                        py_1    conda-forge
numpy                     1.16.2           py37h7e9f1db_0
numpy-base                1.16.2           py37hde5b4d6_0
openssl                   1.1.1b               h7b6447c_1
packaging                 19.0                     py37_0
pcmsolver                 1.2.1            py37h142c950_0    psi4
pint                      0.9                      py37_2    psi4
pip                       19.0.3                   py37_0    conda-forge
pluggy                    0.9.0                      py_0    conda-forge
psi4                      1.3+20e5c7e      py37ha707250_1    psi4
psutil                    5.5.1            py37h14c3975_0    conda-forge
py                        1.8.0                      py_0    conda-forge
py-cpuinfo                4.0.0                      py_0    conda-forge
pycparser                 2.19                     py37_0
pydantic                  0.20.1                   py37_0    conda-forge
pygments                  2.3.1                    py37_0
pyopenssl                 19.0.0                   py37_0
pyparsing                 2.3.1                    py37_0
pysocks                   1.6.8                    py37_0
pytest                    4.3.0                    py37_0    conda-forge
pytest-cov                2.6.1                      py_0    conda-forge
python                    3.7.1             h381d211_1002    conda-forge
pytz                      2018.9                   py37_0
pyyaml                    3.13            py37h14c3975_1001    conda-forge
qcelemental               0.3.1                      py_0    conda-forge
qcengine                  0.6.2+0.gdcd468b.dirty           dev_0    <develop>
readline                  7.0               hf8c457e_1001    conda-forge
requests                  2.21.0                   py37_0
setuptools                40.8.0                   py37_0    conda-forge
simint                    0.7                  h642920c_1    psi4
six                       1.12.0                py37_1000    conda-forge
snowballstemmer           1.2.1                    py37_0
sphinx                    1.8.4                    py37_0
sphinx-automodapi         0.10                     pypi_0    pypi
sphinx_rtd_theme          0.4.2                    py37_0
sphinxcontrib             1.0                      py37_1
sphinxcontrib-websupport  1.1.0                    py37_1
sqlite                    3.26.0            h67949de_1001    conda-forge
tk                        8.6.9             h84994c4_1000    conda-forge
urllib3                   1.24.1                   pypi_0    pypi
wheel                     0.33.1                   py37_0    conda-forge
xz                        5.2.4             h14c3975_1001    conda-forge
yaml                      0.1.7             h14c3975_1001    conda-forge
zlib                      1.2.11            h14c3975_1004    conda-forge

[sjayellis]

I updated my dftd3 and the two tests are passing. Thanks!

pytest qcengine/tests/test_standard_suite.py::test_compute_energy
====================================================================================== test session starts =======================================================================================
platform linux -- Python 3.7.1, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /c/Users/samel/Desktop/MolSSI/QCArchive/QCEngine, inifile: setup.cfg
plugins: cov-2.6.1
collected 4 items

qcengine/tests/test_standard_suite.py ..ss                                                                                                                                                 [100%]

============================================================================== 2 passed, 2 skipped in 1.57 seconds ===============================================================================

pytest qcengine/tests/test_standard_suite.py::test_compute_gradient
====================================================================================== test session starts =======================================================================================
platform linux -- Python 3.7.1, pytest-4.3.0, py-1.8.0, pluggy-0.9.0
rootdir: /c/Users/samel/Desktop/MolSSI/QCArchive/QCEngine, inifile: setup.cfg
plugins: cov-2.6.1
collected 4 items

qcengine/tests/test_standard_suite.py ..ss                                                                                                                                                 [100%]

============================================================================== 2 passed, 2 skipped in 1.74 seconds ===============================================================================

[loriab]

Great! Thanks for testing. The difference is that _0 was built as a static executable, whereas _1 links to some system libraries dynamically. I may publish an _2 that just adjusts the conda deps. Please reopen if it should ever break for you again.