General API for preparing IOData object before dumping

[tovrstra]

Motivation

At the moment, it is mostly assumed that an IOData instance contains all the right attributes in the right form before it is passed on to a dump_one call. Some file formats (WNF, WFX and potentially also FCHK) modify the IOData object to become compatible with the format. Typical modifications include:

• WFX/WFN: converting basis to Cartesian functions (with transformation of the MO coeffs)
• WFX/WFN: decontracting the basis (with transformation of the MO coeffs)
• FCHK: convert (natural) orbitals to density matrix, recontraction of basis sets.
• In general, possibly too far fetched, reverse-engineering contractions from WFN/WFX files.

This is problematic for several reasons:

• It makes dump_one functions long and complicated.
• Users may not be aware of the conversion taking place, which may result in loss of information.
• It may sometimes be of interest to disable conversions, e.g. when they are optional or when the user does not want any conversions (and prefers an exception to be raised instead). The latter is typical when dealing with conversions of large data sets, where data preservation is desirable and unintended loss of information due to conversion is not wanted.
• Some of the current conversions introduce redundant data, which results in inefficient use of storage.

See also:

• Dump fchk #164 (comment)
• Improve treatment of occuptions numbers of restricted orbitals #252 (review)

Proposal

• Add an optional prepare_dump function to the fileformat modules. If present, it takes and IOData instance as argument, and returns a potentially modified one. The given IOData instance is not modified. An option allow_changes=False should be added, to allow disabling any conversion. If this flag is set to False and the file cannot be written without conversion, an exception is raised. If this flag is set to True, a warning is emitted when a conversion is applied.

• The dump_one and dump_many in the file formats functions call the new prepare_dump function before dumping.

• Add a allow_changes=False option to the dump_one and dump_many functions in the file formats modules. This is passed on to the prepare_dump function. The dump functions return the potentially modified IOData instance(s).

• Add a allow_changes=False option to the dump_one and dump_many functions in the module iodata.api. This is passed on to the dump functions of the selected file format. Also these dump functions return the potentially modified IOData instance(s).

• Factor out some of the reusable utility functions to modify the IOData object, e.g. manipulations of basis set and corresponding changes to MO coefficients.

• Add an option to the script iodata-convert to enable or disable modifications before dumping.

• Add basic sanity check to dump_one and dump_many that required attributes are not None before creating a file. Such missing attributes will raise an error, and may result in overwriting the output with an empty file, which is never useful and may ocassionally lead to data loss. This type of pre-flight check could be added to prepare_dump, but it is better to write one general implementation for all file formats, so it is always checked.

TODO list

• Validate that required fields are present in dump_one and dump_many. In dump_one, this can be done before even creating the file. In dump_many this check is possible on the first frame before creating a file, not for later ones. See Check required attributes before dumping to a file #337
• Implement a light version of the prepare_dump API and use it for validity checks: JSON, FCHK, WFX, WFN, Molden, Molekel. This does not add any actual conversion yet. (These will be implemented in later pull requests.) See Add light version of prepare_dump API #344
• Split FileFormatError into LoadError and DumpError and update formats modules to use these consistently. The current use of exceptions in the formats module is not consistent. Idem for FileFormatWarning. See Clean up exceptions and split FileFormatError in LoadError and DumpError #345
• Replace lit.error by raising LoadError directly and extend LoadError with the logic in lit.error. Update contributor guide accordingly. See Replace lit.error by raise LoadError #348
• Replace lit.warn by direct warnings.warn() using an improved LoadWarning class that contains the logic now implemented in lit.warn(). Also extend DumpWarning with file or filename argument, like LoadWarning. Do not use lit.warn Explain LoadWarning in the contributor guide. See Further improve error and warning infrastructure #349
• Extend DumpError with file or filename argument, like LoadError.Directly subclass exceptions from Exception instead of ValueError. (The latter improves testing when using pytest.raises.) See Further improve error and warning infrastructure #349
• Convert MOs to unrestricted if format does not support occs_aminusb: WFN, WFX, Molden, Molekel. Complete prepare_dump API and apply to occs_aminusb + cleanups #352
• Replace catch-all constructs like warnings.catch_warnings() in unit tests by more specific warnings. See Remove useless catch_warnings #353
• Make optional arguments in iodata.api mandatory keywords, by inserting *, in the argument list. See Require optional arguments of iodata.api functions to be specified with keywords #355
• Rename argument iodata in iodata.api to data for consistency with the rest of the code. See Do not use iodata as variable name #356
• Turn Shell attributes into arrays (now lists) with converter functions, in analogy to IOData attributes. Convert Shell attributes to arrays #371
• Move convert_* functions from basis and orbitals to convert module, which becomes the lower-level analog of the prepare module. It does similar things, but without the context of dumping data to files. Move convert functions to new convert module #372
• Write a prepare function to segment a basis before dumping. This can be used by the following formats: Molden, WFN, WFX, FCHK (except for SP shell), Molekel. There should be an option to leave SP shells in place while segmenting all others. At the same time, fix the third point in Clarifications and documentation improvements in iodata.basis #256. See Add prepare_segmented function #373
• Write prepare function to sort shells by center. (Molden and Molekel assume this.)
• Decontract basis and convert MOs in prepare_dump for WFN and WFX. This would also fix Decontraction of basis sets #258.
• Convert MOs to Cartesian basis in prepare_dump for WFN and WFX. This would also fix Conversion to Cartesian basis sets #259.
• Make all convert functions consistent: when no changes are needed, they return reference to the input objects.
• Add --allow-changes option to command-line interface. See Add command-line option --allow-changes to iodata-convert #374
• Extend the contributor guide with the following:
  • Explain how to raise errors and warnings in dump_one
  • Explain how to write a prepare_dump function, and how to raise errors and warnings
• Split getting started into four pages: command-line, loading, dumping and input writing, as also mentioned in Extend the examples in getting_started.rst and run them as unit tests #210. See Split the getting started docs #351
• Expand getting started page on command-line usage to illustrate the --allow-changes option.
• Expand getting started page on dumping, to illustrate the allow_changes keyword argument.

[tovrstra]

Another example of required conversion is discussed in #252: many formats do not support restricted orbitals with "unrestricted occupation numbers". In this case, the orbitals need to be converted to unrestricted form to be able to write a file.