allow and prefer simple-dftd3 and mctc-gcp

[loriab]

Description

User API & Changelog headlines

• Upstream maintained and developed software for Grimme empirical dispersion corrections is now interfaced. The longstanding slight forks maintained by Psi4 folks still work and will be maintained until there's a reason not to. All are still run through QCEngine. Psi4 chooses automatically based on what's detected, so no change to input files needed. Package names and locations are a little different -- see table below.

package	provides	pre-PR	post-PR
D3
psi4::dftd3	bin/dftd3	preferred	works
conda-forge::dftd3-python	import dftd3	nyi	preferred
(dep) conda-forge::simple-dftd3	bin/simple-dftd3
D4
psi4::dftd4	bin/dftd4, import dftd4	preferred	works
conda-forge::dftd4-python	import dftd4	nyi	preferred
(dep) conda-forge::dftd4	bin/dftd4
GCP
psi4::gcp	bin/gcp	preferred	works
conda-forge::gcp-correction	bin/mctc-gcp	nyi	preferred

• Capabilities changed slightly between dftd3 and s-dftd3. In particular, the former can also do -D2 and the latter can do 3-body -D3 in the same call as 2-body -D3. All Psi4 calls will continue to do only 2-body -D3 as default (regardless of dftd3 or s-dftd3 engine). That is, -d3 is still an alias to -d3zero which is now an alias to a new extension -d3zero2b, which can now be given explicitly disallow 3-body as a tweakable parameter (internally, s9:=0.0). When s-dftd3 is the engine, another set of new extensions, e.g., -d3atm alias of -d3zeroatm turns on 3-body (s9=1.0) and allows user tweaks. This latter is the same behavior as -D4, which turns on 3-body by default. If this seems confusing, state what calc you want — e.g., energy("b3lyp-d3atm") — and Psi4 will figure out if you have the right engine to do the job.
• Previous to QCEngine v0.26.0 (required by this Psi4 PR), the pairwise dispersion analysis returned by executable/classic/psi4-channel dftd3 was scaled differently from that by s-dftd3 and dftd4 (any channel). This has been fixed. However, one must be consistent about QCEngine/Psi4/fsapt.py versions. A consistent set are QCEngine >=0.26.0 and Empirical_Disp.dat file generated in the course of energy("fisapt0-d") by this PR or later Psi4 (approx. v1.7.0 Psi4) and script fsapt.py released with Psi4 v1.7.0 or later. This is semi-enforced since v1.7 requires v0.26 (fsapt.py is a free agent). Another consistent set is QCEngine <0.26.0 and Psi4 and fsapt.py <v1.7. Mixing old Psi4 or fsapt.py with new QCEngine and classic dftd3 can yield wrong fisapt analysis, and this isn't trapped.
• psi4.core.Molecule.run_dftd3 and qcdb.Molecule.run_dftd3 don't work with s-dftd3. Please file an issue if you really want this capability. run_gcp will use classic gcp or mctc-gcp interchangeably, whichever you have available.

Dev notes & details

• first look at _engine_can_do and observe that compared to good ol' dftd3, s-dftd3 (aka simple-dftd3) can't do d2 and by default does d3 variants with ATM dispersion built in (s9=1.0)
• in contrast, mctc-gcp is a drop-in replacement for good ol' gcp (probably b/c I stuck with cmdline and didn't write an elaborate interface on top of it :-)
• this goes with adapt s-dftd3 harness for psi4 MolSSI/QCEngine#385
• I didn't think we could just drop the most popular psi4 addon and demand replacement, so all of dftd3/s-dftd3/gcp/mctc-gcp work with psi4. s-dftd3 and mctc-gcp are preferred if present
• need docs
• status (special qcng = v0.26.0 unreleased at time of writing)
  • PR psi4 with special qcng and dftd3 and gcp and psi4-channel dftd4 all work
  • PR psi4 with special qcng and s-dftd3 and mctc-gcp and c-f-channel dftd4 all work except for test_dftd3_mp2d
  • old psi4 with special qcng doesn't work at all -- WIP now works
  • old psi4 running fisapt-d then fsapt.py with special qcng gives wrong answer for emp disp
• run_dftd3 is used a lot in the tests as a qcengine wrapper around psi4 molecules. because dftd3 and s-dftd3 handle defaulting so differently, I can't easily make the fn call either/or. I'm tempted to (a) drop the capability or (b) have a run_sdftd3 that is a thin wrapper like run_dftd4 but won't be a drop-in replacement or (c) deprecate run_dftd3/run_dftd4/run_gcp in favor of a run_dispersion that hooks up to the psi EmpiricalDispersion class and can provide a uniform interface (same logic for disp level hints, tweaked parameters, etc.

Checklist

• Tests added for any new features
• All or relevant fraction of full tests run. For development, I've had two objdirs with two conda envs, one with psi4 disp packages and one with c-f disp packages, and run tests on each.

Status

• Ready for review
• Ready for merge

[hokru]

Worth checking if ATM should stay off. Can't recall what the paper recommended.

[loriab]

According to eqns 42, 44, 52 in https://psi-k.net/download/highlights/Highlight_144.pdf , ATM is off, on, on for HF-3c, PBEh-3c, B97-3c. But I also realize I don't know if gCP could be covering that.

For the moment, this change is preserving the previous behavior. So there's still the opportunity for a deliberate fix to match the literature later.

[hokru]

Agree with preserving the old behavior for now.

[hokru]

If it stays d3bj then it depends on the engine if the 3-body term is included? Slightly confused about the aliases on the initial reading.

[loriab]

The ATM default for classic dftd3 is off and s-dftd3 is on, so that's what a raw qcengine call would do since qcengine's mandate (approximately) is pass-through data rearrangement. Psi4's goal is an even interface over all the disp programs, so it pre-solves the parameters and makes a customized request to either qcengine harness. That way, the Psi4 default is 2-body, no matter whether dftd3 or s-dftd3 is available. If one requests b3lyp-d3bjatm and only classic dftd3 is around, it'll balk. Somewhere I'll need to document that better.

[loriab]

Reading this again, I think I understand the question. If the user had an input with d3bj, no need to edit, as d3bj did and now continues to default to 2body-only, regardless of engine. I changed the type here (1) to emphasize that d3bj2b is the fundamental type rather than d3bj and (2) these definitions, as opposed to input files, don't get aliases applied so they need the fundamental type (possibly I relaxed this).

[hokru]

I asked because of your point (2). We developers then need to pay attention to the correct type.

However users with old input files might get slightly different results if they use a custom functional. Likely a very very small number of users. Those probably read the changelog.

okay to not relax this.

[loriab]

I've checked with the dft-custom test, and one can use the aliases in custom functionals, so old inputs will work and answers not change. Apparently the fundamental type requirement was an earlier version of the code.

[JonathonMisiewicz]

I appreciate the many tests!

Looks good overall, but I defer to Holger on the points he raised.

[loriab]

I appreciate the many tests!

I only hope they're enough. there's 5 disp programs out there governed by 4 sets of logic and about 3 sets of defaults. :-)

[hokru]

Agree with preserving the old behavior for now.

[hokru]

I asked because of your point (2). We developers then need to pay attention to the correct type.

However users with old input files might get slightly different results if they use a custom functional. Likely a very very small number of users. Those probably read the changelog.

okay to not relax this.