Separate preprocessing, MPI invocation, and post-processing for codes which use MPI tasks

[WardLT]

Is your feature request related to a problem? Please describe.
The motivation for this feature request is being able to use more sophisticated systems for launching MPI tasks in large scale workflows (e.g., RADICAL cybertools). Systems such as RCT for do not rely on calling mpiexec directly from the shell (something about passing MPI communicators around) and using such methods is inhibited by the MPI call in QCEngine being buried deep inside of a call stack.

The logic is pretty much there already in NWChemHarness's build_inputs and parse_outputs functions. So, we just need to document that's what they can actually be used for splitting up a call into different steps if that's true for enough codes.

Describe the solution you'd like
A standard interface to generating input file content

harness = NWChemHarness()

input_files: dict = harness.build_inputs(atomic_input)

[... workflow engine stuff ...]

result = harness.parse_outputs(path: Union[Path, str], atomic_input)

that I'm confident in!

Describe alternatives you've considered
Using parse_outputs and build_inputs to break up the use of the program harness and building NWChem-specific logic.

Additional context

I'm going to start integrating NWChem into these "MPI processes are special" workflow engines (e.g., RCT, Balsam) and will update this issue with my progress.

[loriab]

Is it localized in the harness execute fn (e.g., https://github.com/MolSSI/QCEngine/blob/master/qcengine/programs/nwchem/runner.py#L258-L271) that needs to change for RADICAL? One thing I've done is inherit the harness and replace the individual build_input/execute/parse_output.

The build/exe/parse=compute is pretty stable for the non-API harnesses, though I haven't done a survey.

[WardLT]

Yea, I think it's just the execute function that is different for how to invoke an MPI code with RCT. I'm working with them on a simpler example and will know more once I finish that.