Orca Harness. #178

muammar · 2019-11-22T18:04:42Z

Description

Orca Harness. It goes along with MolSSI/QCElemental#186.

Status

WIP
Ready to go

* master: (61 commits) Config: Changes new programs from job to task config Config: JobConfig -> TaskConfig Config: Adds nodes to config entos: Black reformat of entos.py and added @using_entos mark PR comments PR comments: TYPE_CHECKING, formatting, reduce redundant code entos: Added entos_policy to the input file entos: Updated commit for qcengine_records_commit hessparse and compute: Forgotten imports entos: Begun adding support for xtb in the entos Harness entos: Removed soon to be deprecated option entos: Added entos to the test_standard_suite_hf and did some minor cleanup in the entos Harness entos: Starting to add HF command to entos Fixed typo in error name Multiple changes based on review Gamess: replace print statements with logging Entos: add hessian and xtb support Parse named properties out of NWChem output Get more of the relevant fields from output Removed unncessary "pass" statement ...

lgtm-com · 2019-12-03T19:16:25Z

This pull request introduces 3 alerts when merging 2657c79 into f9fea80 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Unused import
1 for Variable defined multiple times

codecov · 2019-12-04T23:40:22Z

Codecov Report

Merging #178 into master will decrease coverage by 1.91%.
The diff coverage is 22.13%.

lgtm-com · 2019-12-04T23:56:37Z

This pull request introduces 5 alerts when merging 97c63cc into f9fea80 - view on LGTM.com

new alerts:

2 for Unused local variable
2 for Unused import
1 for Variable defined multiple times

muammar · 2019-12-05T00:00:09Z

This pull request introduces 5 alerts when merging 97c63cc into f9fea80 - view on LGTM.com

new alerts:

2 for Unused local variable

2 for Unused import

1 for Variable defined multiple times

yeah I know, good bot

- Total energy is returned when using DFT. - `commands` now contains the full PATH to the orca binary. That is important to run the binary in paralell mode. - Removal/commenting of unused stuff.

lgtm-com · 2019-12-05T18:15:38Z

This pull request introduces 2 alerts when merging e6c00c2 into f9fea80 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

lgtm-com · 2019-12-05T18:41:24Z

This pull request introduces 2 alerts when merging ca4f623 into f9fea80 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

lgtm-com · 2019-12-05T19:17:12Z

This pull request introduces 2 alerts when merging bd28893 into f9fea80 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

lgtm-com · 2019-12-09T23:23:02Z

This pull request introduces 2 alerts when merging d257a35 into 16194de - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

* upstream/master: Qchem: SCF-> Total energy in the final basis set for correctly catching -D, etc. Qchem: PR comments Testing: Finishes up using conversion on several straglers Testing: Applies new using strategy to all files Testing: New using strategy Psi4: Fixes issue with HF3c execution PR comments PR comments: parse logfile for fields not available in QCSCR also, fix dipole parsing nwc: add hessian file; minor testing updates (MolSSI#177) PR comments: use qcel's NUMBER regex also, blacken and isort fix test until next qcel version PR comments: added ability to ingest logfile+QCSCR Blacken QCEngine PR comments: rename input, refactor NUMBER into regex snippet file Update QCENGINE_RECORDS_COMMIT Update QCENGINE_RECORDS_COMMIT Qchem: add tests for log parser for archival data Qchem: add log parser for archival data

lgtm-com · 2019-12-09T23:45:37Z

This pull request introduces 2 alerts when merging 1e96ea0 into 9f5fba2 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

- Cleaned up unneeded comments. - Added support for HF method. - Parallel computation support using %pal. - Black beautification.

lgtm-com · 2019-12-10T00:22:38Z

This pull request introduces 2 alerts when merging 006afb8 into 9f5fba2 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

lgtm-com · 2019-12-10T01:20:00Z

This pull request introduces 2 alerts when merging 5620a66 into 9f5fba2 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

* master: (21 commits) Docs: Changelog 0.13.0 extension Psi4: Ensures output data for psi4, fixes MolSSI#176 Docs: Adds v0.13.0 changelog Lint: Fixes GHA routine Testing: Canonicalizes from testing, closes MolSSI#193 Lint: Adds a GHA action to check for Black GitHub: Updates tempaltes to QCArchive standard Lint: Blackens code base Addresses MolSSI#186 Improved MPI support Clarified documentation Improved documentation and type checking for node configuration Added ability to forward output from popen Documentation for configs needed for node-parallel tasks Removed unused import Bug fix: Use integer division not floating point Read from both stderr and stdout to prevent buffer deadlock Bug fix: Memory is per MPI rank Bug fix: Missing task configuration when making mpi command Improved warnings for when multi-node jobs are misconfigured ...

lgtm-com · 2019-12-11T22:25:44Z

This pull request introduces 2 alerts when merging 4a125bc into 30030d5 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

* master: NWCHEM: parse dispersion_correction output from psivar/ fix bug Read Dispersion Energy from nwchem output file and add it to psivar Update harvester.py

lgtm-com · 2019-12-18T19:04:09Z

This pull request introduces 2 alerts when merging 9dfc4e6 into fcd6f3d - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Variable defined multiple times

* master: (29 commits) CI for NWChem (MolSSI#212) Make sure input extras tags get to the output Added NWChem to the canonical harness tests Calls to rtdb must use all threads Remove assumption that NWChem uses original order of atoms Fixed test for cores_per_rank Correctly determine memory sizes, when MPI should be used Account for cores_per_rank when computing total ranks Added a more difficult test for the nwchem hessian Read hessian array in proper order Use pep8 variable names Rotate gradients and hessian to match input molecule Linted with black Added Hessian support and improved gradient accuracy Allow for methods that are combinations of several functionals psi4 windows version and outfile psiapi mode (MolSSI#201) NWCHEM: added keyword dft to xc_functionals list. If method == "dft" (MolSSI#204) entos: Updated QCER hash and program_overview.rst black reformatting entos: AO reordering added to unrestricted wavefunction properties ...

- Removed Molpro references. - Removed restricted/unrestricted case guessing.

muammar · 2020-01-09T19:31:10Z

@dgasmith would it be possible to have the first review to know what steps to take forward? The only check not passing right now is the CI. I would not know how to do this for Orca because it has a very particular way to be installed :(.

loriab

Testing Orca with CI will indeed be tricky since one needs a license. But before that problem, it'd be good to add some tests locally so you know this is running. You can add it to here (https://github.com/MolSSI/QCEngine/blob/fd086f628bf9e18e1b6ea1e8e78a6623f209b934/qcengine/tests/test_harness_canonical.py) which smoke tests operation. There's also the standard_suite tests, which see if answers match other programs. And whatever other independent tests you like, as some other programs have. To address the CI situation, we'll probably have you add to the QCEngingRecords repo, which tests input writing and parsing. But since you have Orca locally, tests in this repo are the place to start.

loriab · 2020-01-09T20:00:23Z

qcengine/programs/orca.py

+ # caseless_keywords = {k.lower(): v for k, v in input_model.keywords.items()}
+
+ # Write the geom
+ xyz_block = input_model.molecule.to_string(dtype="orca", units="Angstrom")


there isn't an orca dtype at the moment. you can add one here or work with me so I can write one.

also, running in bohr is strongly preferred since atomic units don't change with codata revisions.

Along with this PR there is another PR for QCElemental. I will create it soon. I will change to bohr instead. Let me research about it.

qcengine/programs/orca.py

dgasmith

Overall, this is looking really good!

qcengine/programs/orca.py

- `conversion_factor()` function used to convert from eV to Hartree. - cclib is just imported when invoking `.parse_output()`. - `.get_gradient()` renamed to `._get_gradient()` so that it becomes a private function.

lgtm-com · 2020-01-10T17:48:11Z

This pull request introduces 1 alert when merging 21ae056 into e370e86 - view on LGTM.com

new alerts:

1 for Illegal raise

* origin/master: simplify better error message from get_program, closes MolSSI#213 pass for psi w/o --module prep env for psithon or psiapi, get the other for free psithon _or_ psiapi for p4

lgtm-com · 2020-01-16T18:22:39Z

This pull request introduces 1 alert when merging adba913 into d31f65b - view on LGTM.com

new alerts:

1 for Illegal raise

lgtm-com · 2020-01-17T04:55:26Z

This pull request introduces 2 alerts when merging e192143 into d31f65b - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Illegal raise

* master: (40 commits) Docs: Minor changelog tweaks based on comments remove the unnecessary if else blocks Docs: Cleans up code blocks OpenMM: Switches method/basis as discussed Docs: Adds notes on molecular mechancs programs Docs: Patches up doc errors Docs: Adds OpenMM to the overview Docs: Adds 0.14.0 changelog fix logic only check Nalpha ==Nbeta if already in psivar OpenMM: Now requires openforcefield as well make psi4 harness safe for psiapi file cleaning CI: Re-enables TorchANI CI testing CI: Removes duplicate RDKit build OpenMM: Adds tests for other openff, ffs Psi4: Bumps dev branch to 500 Testing: Patches up tests after QCElemental sparse molecule changes OpenMM: Fixes forces and adds an optimization example OpenMM: Adds openmm tests to canonical tests OpenMM: Simplifies method/basis in accordance with MolSSI#192 OpenMM: Testing now depends on rdkit ...

lgtm-com · 2020-02-07T02:36:52Z

This pull request introduces 2 alerts when merging d505c4b into c8ae155 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Illegal raise

lgtm-com · 2020-02-07T17:15:56Z

This pull request introduces 2 alerts when merging 2b69b56 into c8ae155 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Illegal raise

* master: fixed format add procedure for pyberny geometry optimizer Add MDI version check Linted mdi_server.py Update to support MDI v1.1.3 added optional param for execute: exit_code add conda-envs readme fixing my h20v2 mistake add homo lumo test added homo lumo test caputure +- 0 and exponent all at once, remove if-else format comment for clarity if else for for +/- exponent

lgtm-com · 2020-03-13T07:06:49Z

This pull request introduces 2 alerts when merging 2958358 into 4267201 - view on LGTM.com

new alerts:

1 for Unused local variable
1 for Illegal raise

yueyericardo · 2023-08-24T18:42:24Z

Hi, what is the status of this PR. Could we merge it?
Orca was used for the original ANI datasets calculation, psi4 energy does not match with orca.

yueyericardo · 2023-08-24T23:31:53Z

Just for your information, I manually added this PR into my local fork. And orca is working correctly on my end.

yueyericardo · 2023-08-25T07:21:27Z

Hi, I'm not very familiar with the QCArchive codebase and found difficulties when run orca with qcarchive. The problem is that parsl is good for multi-threading software, but it does not work well with software like orca that is compiled against OpenMPI.

I managed a workaround by adjusting the configuration as follows. From Parsl's perspective, this ensures:
A single task_command for each worker
56 mpi_ranks (equivalent to ntasks in SLURM).
Only 1 cores_per_worker in Parsl's view, translating to 1 cpus_per_task in SLURM.

# manager.yaml
common:
 adapter: parsl
 tasks_per_worker: 1 
 cores_per_worker: 1  # this is a workaround for parsl because it determine cpus_per_task = cores_per_node / tasks_per_node, we need cpus_per_task=1 
 memory_per_worker: 192 
 max_workers: 20 
 scratch_directory: "$SCRATCH"

cluster:
 node_exclusivity: True
 scheduler: slurm
 scheduler_options: 
  - --ntasks-per-node=56  # this is because orca uses multi-processing instead of multi-threading

however, I would now need to mannually hack the orca.py to set the number of process

            # parallel = "%pal nproc {} end".format(config.ncores)
            config_ncores = 56
            parallel = "%pal nproc {} end".format(config_ncores)

because the launch commandprocess_worker_pool.py --max_workers=1 -a login1.frontera.tacc.utexas.edu -p 0 -c 1 set number of core equals 1 now......

Any guidance or improvements to fix this issue would be appreciated!
Thanks!

loriab

Thanks for pursuing this. Please go ahead and rebase so that CI can run.

loriab · 2023-08-30T04:03:56Z

qcengine/programs/orca.py

+ _output = output["stdout"].split()
+
+ for index, line in enumerate(_output):
+ if "Version" in line:
+ found = True
+ if found:
+ version = _output[index + 1]
+ break


Shouldn't this block be indented so that it only runs after success, output = execute(?

loriab · 2023-08-30T04:32:32Z

qcengine/programs/orca.py

+
+ # Write appropriate driver call
+ if input_model.driver == "energy":
+ input_file.extend(energy_call)


Is energy_call var ever non-empty?

loriab · 2023-08-30T04:35:37Z

qcengine/programs/orca.py

+ input_file.append("! {} {}".format(input_model.model.method, input_model.model.basis))
+ input_file.append(xyz_block)
+
+ elif input_model.model.method.upper() in self._dft_functionals or self._hf_methods: # DFT case


I don't think this is going to catch self._hf_methods

>>> aa = [1, 2, 3, 4] >>> bb = [5, 6, 7, 8] >>> 2 in aa or bb True >>> 7 in aa or bb [5, 6, 7, 8]

loriab · 2023-08-30T05:25:14Z

qcengine/programs/orca.py

+
+ def parse_output(self, outfiles: Dict[str, str], input_model: "AtomicInput") -> "AtomicResult":
+ try:
+ import cclib


It's preferred to detect needed deps in the found function so that compute has the best chance to run cleanly. There's an example with networkx in nwchem/runner.py .

muammar added 2 commits November 22, 2019 10:03

Small advances to Orca Harness.

6b26a41

Cleaned up print statements and output file is parsed by cclib.

97c63cc

General improvements.

e6c00c2

- Total energy is returned when using DFT. - `commands` now contains the full PATH to the orca binary. That is important to run the binary in paralell mode. - Removal/commenting of unused stuff.

Cleaning up print statement and addition of more properties.

ca4f623

Removed commented files.

bd28893

Gradient is returned as result.

d257a35

muammar added 2 commits December 9, 2019 15:35

Current energy added to gradient extras.

e9ff4a6

General improvements

006afb8

- Cleaned up unneeded comments. - Added support for HF method. - Parallel computation support using %pal. - Black beautification.

Fixed gradient dimensionality.

5620a66

Merge branch 'master' into orca

9dfc4e6

* master: NWCHEM: parse dispersion_correction output from psivar/ fix bug Read Dispersion Energy from nwchem output file and add it to psivar Update harvester.py

muammar added 4 commits January 9, 2020 10:46

Remove unneeded code.

ec3702e

General improvements:

426c36a

- Removed Molpro references. - Removed restricted/unrestricted case guessing.

Pyflakes clean.

2624581

loriab reviewed Jan 9, 2020

View reviewed changes

muammar mentioned this pull request Jan 9, 2020

Addition of Orca dtype MolSSI/QCElemental#186

Merged

2 tasks

Bohr by default.

0e3d699

dgasmith reviewed Jan 10, 2020

View reviewed changes

qcengine/programs/orca.py Outdated Show resolved Hide resolved

qcengine/programs/orca.py Show resolved Hide resolved

qcengine/programs/orca.py Outdated Show resolved Hide resolved

qcengine/programs/orca.py Outdated Show resolved Hide resolved

General improvements following reviews.

21ae056

- `conversion_factor()` function used to convert from eV to Hartree. - cclib is just imported when invoking `.parse_output()`. - `.get_gradient()` renamed to `._get_gradient()` so that it becomes a private function.

muammar added 2 commits January 16, 2020 10:06

More changes based on review.

c74b32d

Merge remote-tracking branch 'origin/master' into orca

adba913

* origin/master: simplify better error message from get_program, closes MolSSI#213 pass for psi w/o --module prep env for psithon or psiapi, get the other for free psithon _or_ psiapi for p4

muammar added 2 commits January 16, 2020 20:45

First steps to post-HF methods.

d160f3d

Remove print.

e192143

muammar changed the title ~~WIP: Orca Harness.~~ Orca Harness. Feb 7, 2020

muammar added 2 commits February 7, 2020 09:01

Added Orca canonical example.

892b502

Registered orca in testing module.

2b69b56

leeping mentioned this pull request Sep 15, 2020

ORCA support request leeping/geomeTRIC#122

Closed

jthorton mentioned this pull request Jun 17, 2022

Gaussian Input format qubekit/QUBEKit#317

Closed

loriab reviewed Aug 30, 2023

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Orca Harness. #178

Orca Harness. #178

muammar commented Nov 22, 2019 •

edited

lgtm-com bot commented Dec 3, 2019

codecov bot commented Dec 4, 2019 •

edited

lgtm-com bot commented Dec 4, 2019

muammar commented Dec 5, 2019

lgtm-com bot commented Dec 5, 2019

lgtm-com bot commented Dec 5, 2019

lgtm-com bot commented Dec 5, 2019

lgtm-com bot commented Dec 9, 2019

lgtm-com bot commented Dec 9, 2019

lgtm-com bot commented Dec 10, 2019

lgtm-com bot commented Dec 10, 2019

lgtm-com bot commented Dec 11, 2019

lgtm-com bot commented Dec 18, 2019

muammar commented Jan 9, 2020

loriab left a comment

loriab Jan 9, 2020

muammar Jan 9, 2020

dgasmith left a comment

lgtm-com bot commented Jan 10, 2020

lgtm-com bot commented Jan 16, 2020

lgtm-com bot commented Jan 17, 2020

lgtm-com bot commented Feb 7, 2020

lgtm-com bot commented Feb 7, 2020

lgtm-com bot commented Mar 13, 2020

yueyericardo commented Aug 24, 2023

yueyericardo commented Aug 24, 2023 •

edited

yueyericardo commented Aug 25, 2023 •

edited

loriab left a comment

loriab Aug 30, 2023

loriab Aug 30, 2023

loriab Aug 30, 2023

loriab Aug 30, 2023

Orca Harness. #178

Are you sure you want to change the base?

Orca Harness. #178

Conversation

muammar commented Nov 22, 2019 • edited

Description

Status

lgtm-com bot commented Dec 3, 2019

codecov bot commented Dec 4, 2019 • edited

Codecov Report

lgtm-com bot commented Dec 4, 2019

muammar commented Dec 5, 2019

lgtm-com bot commented Dec 5, 2019

lgtm-com bot commented Dec 5, 2019

lgtm-com bot commented Dec 5, 2019

lgtm-com bot commented Dec 9, 2019

lgtm-com bot commented Dec 9, 2019

lgtm-com bot commented Dec 10, 2019

lgtm-com bot commented Dec 10, 2019

lgtm-com bot commented Dec 11, 2019

lgtm-com bot commented Dec 18, 2019

muammar commented Jan 9, 2020

loriab left a comment

Choose a reason for hiding this comment

loriab Jan 9, 2020

Choose a reason for hiding this comment

muammar Jan 9, 2020

Choose a reason for hiding this comment

dgasmith left a comment

Choose a reason for hiding this comment

lgtm-com bot commented Jan 10, 2020

lgtm-com bot commented Jan 16, 2020

lgtm-com bot commented Jan 17, 2020

lgtm-com bot commented Feb 7, 2020

lgtm-com bot commented Feb 7, 2020

lgtm-com bot commented Mar 13, 2020

yueyericardo commented Aug 24, 2023

yueyericardo commented Aug 24, 2023 • edited

yueyericardo commented Aug 25, 2023 • edited

loriab left a comment

Choose a reason for hiding this comment

loriab Aug 30, 2023

Choose a reason for hiding this comment

loriab Aug 30, 2023

Choose a reason for hiding this comment

loriab Aug 30, 2023

Choose a reason for hiding this comment

loriab Aug 30, 2023

Choose a reason for hiding this comment

muammar commented Nov 22, 2019 •

edited

codecov bot commented Dec 4, 2019 •

edited

yueyericardo commented Aug 24, 2023 •

edited

yueyericardo commented Aug 25, 2023 •

edited