New user input for grid type (collocated, staggered, hybrid) (#3683)

* Introduce `warpx.grid_type` parameter

* Replace `or` with `||`

* Update examples with new user input syntax

* Fix `if` condition

* Improve error message

* Fix `if` condition

* Fix bugs

* Fix warning

* Fix RZ

* Debugging

* Fix RZ

* Fix bug

* Clean up

* More changes:

- set default algo parameters with hybrid grid
- all hybrid input parameters under warpx name

* Set default field gathering algo for hybrid grids

* Update documentation

Docs/source/developers/qed.rst

@@ -40,6 +40,6 @@ pairs created.
 At most one macroparticle is created per cell per timestep per species, with a weight corresponding to
 the total number of physical pairs created.
 
-So far the Schwinger module requires using ``warpx.do_nodal=1`` or
-``algo.field_gathering=momentum-conserving`` (so that the auxiliary fields are calculated on the nodes)
+So far the Schwinger module requires using ``warpx.grid_type = collocated`` or
+``algo.field_gathering = momentum-conserving`` (so that the auxiliary fields are calculated on the nodes)
 and is not compatible with either mesh refinement, RZ coordinates or single precision.

Docs/source/usage/parameters.rst

@@ -1748,14 +1748,13 @@ Particle push, charge and current deposition, field gathering
 * ``algo.field_gathering`` (`string`, optional)
  The algorithm for field gathering. Available options are:
 
- - ``energy-conserving``: gathers directly from the grid points (either staggered
- or nodal gridpoints depending on ``warpx.do_nodal``).
- - ``momentum-conserving``: first average the fields from the grid points to
+ * ``energy-conserving``: gathers directly from the grid points (either staggered
+ or nodal grid points depending on ``warpx.grid_type``).
+ * ``momentum-conserving``: first average the fields from the grid points to
  the nodes, and then gather from the nodes.
 
- If ``algo.field_gathering`` is not specified, the default is ``energy-conserving``.
- If ``warpx.do_nodal`` is ``true``, then ``energy-conserving`` and ``momentum-conserving``
- are equivalent.
+
+ Default: ``algo.field_gathering = energy-conserving`` with collocated or staggered grids (note that ``energy-conserving`` and ``momentum-conserving`` are equivalent with collocated grids), ``algo.field_gathering = momentum-conserving`` with hybrid grids.
 
 * ``algo.particle_pusher`` (`string`, optional)
  The algorithm for the particle pusher. Available options are:
@@ -1810,7 +1809,7 @@ Maxwell solver: PSATD method
 * ``psatd.nx_guard``, ``psatd.ny_guard``, ``psatd.nz_guard`` (`integer`) optional
  The number of guard cells to use with PSATD solver.
  If not set by users, these values are calculated automatically and determined *empirically* and
- would be equal the order of the solver for nodal grid, and half the order of the solver for staggered.
+ equal the order of the solver for collocated grids and half the order of the solver for staggered grids.
 
 * ``psatd.periodic_single_box_fft`` (`0` or `1`; default: 0)
  If true, this will *not* incorporate the guard cells into the box over which FFTs are performed.
@@ -1967,31 +1966,42 @@ Maxwell solver: macroscopic media
 Grid types (collocated, staggered, hybrid)
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-* ``warpx.do_nodal`` (`0` or `1` ; default: 0)
- Whether to use a nodal grid (i.e. all fields are defined at the
- same points in space) or a staggered grid (i.e. Yee grid ; different
- fields are defined at different points in space)
+* ``warpx.grid_type`` (`string`, ``collocated``, ``staggered`` or ``hybrid``)
+ Whether to use a collocated grid (all fields defined at the cell nodes),
+ a staggered grid (fields defined on a Yee grid), or a hybrid grid (fields
+ and currents are interpolated back and forth between a staggered grid and a
+ nodal grid, must be used with momentum-conserving field gathering algorithm,
+ ``algo.field_gathering = momentum-conserving``).
+
+ Default: ``warpx.grid_type = staggered``.
 
 * ``interpolation.galerkin_scheme`` (`0` or `1`)
  Whether to use a Galerkin scheme when gathering fields to particles.
- When set to `1`, the interpolation orders used for field-gathering are reduced for certain field components along certain directions.
+ When set to ``1``, the interpolation orders used for field-gathering are reduced for certain field components along certain directions.
  For example, :math:`E_z` is gathered using ``algo.particle_shape`` along :math:`(x,y)` and ``algo.particle_shape - 1`` along :math:`z`.
  See equations (21)-(23) of (`Godfrey and Vay, 2013 <https://doi.org/10.1016/j.jcp.2013.04.006>`_) and associated references for details.
- Defaults to `1` unless ``warpx.do_nodal = 1`` and/or ``algo.field_gathering = momentum-conserving``.
+
+ Default: ``interpolation.galerkin_scheme = 0`` with collocated grids and/or momentum-conserving field gathering, ``interpolation.galerkin_scheme = 1`` otherwise.
 
  .. warning::
 
  The default behavior should not normally be changed.
  At present, this parameter is intended mainly for testing and development purposes.
 
-* ``interpolation.field_centering_nox``, ``interpolation.field_centering_noy``, ``interpolation.field_centering_noz`` (default: ``2`` in all directions)
- The order of interpolation used with staggered grids (``warpx.do_nodal = 0``) and momentum-conserving field gathering (``algo.field_gathering = momentum-conserving``) to interpolate the electric and magnetic fields from the cell centers to the cell nodes, before gathering the fields from the cell nodes to the particle positions. High-order interpolation (order 8 in each direction, at least) is necessary to ensure stability in typical LWFA boosted-frame simulations using the Galilean PSATD or comoving PSATD schemes. This finite-order interpolation is used only when the PSATD solver is used for Maxwell's equations. With the FDTD solver, basic linear interpolation is used instead.
+* ``warpx.field_centering_nox``, ``warpx.field_centering_noy``, ``warpx.field_centering_noz`` (`integer`, optional)
+ The order of interpolation used with staggered or hybrid grids (``warpx.grid_type = staggered`` or ``warpx.grid_type = hybrid``) and momentum-conserving field gathering (``algo.field_gathering = momentum-conserving``) to interpolate the electric and magnetic fields from the cell centers to the cell nodes, before gathering the fields from the cell nodes to the particle positions.
+
+ Default: ``warpx.field_centering_no<x,y,z> = 2`` with staggered grids, ``warpx.field_centering_no<x,y,z> = 8`` with hybrid grids (typically necessary to ensure stability in boosted-frame simulations of relativistic plasmas and beams).
+
+* ``warpx.current_centering_nox``, ``warpx.current_centering_noy``, ``warpx.current_centering_noz`` (`integer`, optional)
+ The order of interpolation used with hybrid grids (``warpx.grid_type = hybrid``) to interpolate the currents from the cell nodes to the cell centers when ``warpx.do_current_centering = 1``, before pushing the Maxwell fields on staggered grids.
+
+ Default: ``warpx.current_centering_no<x,y,z> = 8`` with hybrid grids (typically necessary to ensure stability in boosted-frame simulations of relativistic plasmas and beams).
 
-* ``interpolation.current_centering_nox``, ``interpolation.current_centering_noy``, ``interpolation.current_centering_noz`` (default: ``2`` in all directions)
- The order of interpolation used to center the currents from nodal to staggered grids (if ``warpx.do_current_centering = 1``), before pushing the Maxwell fields on staggered grids. This finite-order interpolation is used only when the PSATD solver is used for Maxwell's equations. With the FDTD solver, basic linear interpolation is used instead.
+* ``warpx.do_current_centering`` (`bool`, `0` or `1`)
+ If true, the current is deposited on a nodal grid and then centered to a staggered grid (Yee grid), using finite-order interpolation.
 
-* ``warpx.do_current_centering`` (`0` or `1` ; default: 0)
- If true, the current is deposited on a nodal grid and then centered to a staggered grid (Yee grid), using finite-order interpolation. If ``warpx.do_nodal = 1``, the Maxwell fields are pushed on a nodal grid, it is not necessary to center the currents to a staggered grid, and we set therefore ``warpx.do_current_centering = 0`` automatically, overwriting the user-defined input.
+ Default: ``warpx.do_current_centering = 0`` with collocated or staggered grids, ``warpx.do_current_centering = 1`` with hybrid grids.
 
 Additional parameters
 ^^^^^^^^^^^^^^^^^^^^^
@@ -2031,8 +2041,8 @@ Additional parameters
  Will use the Hybird QED Maxwell solver when pushing fields: a QED correction is added to the
  field solver to solve non-linear Maxwell's equations, according to [Quantum Electrodynamics
  vacuum polarization solver, P. Carneiro et al., `ArXiv 2016 <https://arxiv.org/abs/1607.04224>`__].
- Note that this option can only be used with the PSATD build. Furthermore,
- warpx.do_nodal must be set to `1` which is not its default value.
+ Note that this option can only be used with the PSATD build. Furthermore, one must set
+ ``warpx.grid_type = collocated`` (which otherwise would be ``staggered`` by default).
 
 * ``warpx.quantum_xi`` (`float`; default: 1.3050122.e-52)
  Overwrites the actual quantum parameter used in Maxwell's QED equations. Assigning a
@@ -2856,7 +2866,7 @@ Lookup tables store pre-computed values for functions used by the QED modules.
  Activating the Schwinger process requires the code to be compiled with ``QED=TRUE`` and ``PICSAR``.
  If ``warpx.do_qed_schwinger = 1``, Schwinger product species must be specified with
  ``qed_schwinger.ele_product_species`` and ``qed_schwinger.pos_product_species``.
- Schwinger process requires either ``warpx.do_nodal=1`` or
+ Schwinger process requires either ``warpx.grid_type = collocated`` or
  ``algo.field_gathering=momentum-conserving`` (so that different field components are computed
  at the same location in the grid) and does not currently support mesh refinement, cylindrical
  coordinates or single precision.

Examples/Tests/comoving/inputs_2d_hybrid

@@ -17,7 +17,6 @@ boundary.field_hi = periodic damped
 algo.maxwell_solver = psatd
 algo.current_deposition = direct
 algo.charge_deposition = standard
-algo.field_gathering = momentum-conserving
 algo.particle_pusher = vay
 
 # Order of particle shape factors
@@ -27,7 +26,8 @@ psatd.use_default_v_comoving = 1
 
 warpx.cfl = 1.
 
-warpx.do_nodal = 0
+warpx.grid_type = hybrid
+warpx.do_current_centering = 0
 
 warpx.gamma_boost = 13.
 warpx.boost_direction = z
@@ -41,9 +41,6 @@ warpx.use_filter = 1
 warpx.serialize_initial_conditions = 1
 warpx.verbose = 1
 
-interpolation.field_centering_nox = 8
-interpolation.field_centering_noz = 8
-
 particles.species_names = electrons ions beam
 particles.use_fdtd_nci_corr = 0
 particles.rigid_injected_species = beam

Examples/Tests/maxwell_hybrid_qed/inputs_2d

@@ -9,7 +9,7 @@ geometry.dims = 2
 geometry.prob_lo = -32.e-6 -512.e-6
 geometry.prob_hi = 32.e-6 512.e-6
 amr.max_level = 0
-warpx.do_nodal = 1
+warpx.grid_type = collocated
 warpx.quantum_xi = 1.e-23
 
 #################################

Examples/Tests/nci_psatd_stability/inputs_2d

@@ -38,7 +38,7 @@ algo.particle_shape = 3
 #################################
 particles.species_names = electrons ions
 
-warpx.do_nodal = 1
+warpx.grid_type = collocated
 warpx.use_filter = 1
 
 psatd.nox = 16

Examples/Tests/nci_psatd_stability/inputs_2d_hybrid

@@ -15,7 +15,6 @@ boundary.field_hi = periodic damped
 algo.maxwell_solver = psatd
 algo.current_deposition = direct
 algo.charge_deposition = standard
-algo.field_gathering = momentum-conserving
 algo.particle_pusher = vay
 
 # Order of particle shape factors
@@ -25,7 +24,8 @@ psatd.use_default_v_galilean = 1
 
 warpx.cfl = 1.
 
-warpx.do_nodal = 0
+warpx.grid_type = hybrid
+warpx.do_current_centering = 0
 
 warpx.gamma_boost = 13.
 warpx.boost_direction = z
@@ -39,9 +39,6 @@ warpx.use_filter = 1
 warpx.serialize_initial_conditions = 1
 warpx.verbose = 1
 
-interpolation.field_centering_nox = 8
-interpolation.field_centering_noz = 8
-
 particles.species_names = electrons ions beam
 particles.use_fdtd_nci_corr = 0
 particles.rigid_injected_species = beam

Examples/Tests/nci_psatd_stability/inputs_3d

@@ -75,7 +75,7 @@ psatd.noz = 8
 
 # warpx
 warpx.cfl = 1.
-warpx.do_nodal = 1
+warpx.grid_type = collocated
 warpx.numprocs = 1 1 2
 warpx.use_filter = 1
 warpx.verbose = 1

Examples/Tests/nci_psatd_stability/inputs_avg_2d

@@ -40,7 +40,7 @@ algo.particle_shape = 3
 #################################
 particles.species_names = electrons ions
 
-warpx.do_nodal = 1
+warpx.grid_type = collocated
 warpx.use_filter = 1
 
 psatd.nox = 16

Examples/Tests/nci_psatd_stability/inputs_avg_3d

@@ -38,7 +38,7 @@ algo.particle_shape = 3
 #################################
 particles.species_names = electrons ions
 
-warpx.do_nodal = 1
+warpx.grid_type = collocated
 warpx.use_filter = 1
 
 psatd.nox = 8

Examples/Tests/nci_psatd_stability/inputs_rz

@@ -34,7 +34,7 @@ algo.particle_shape = 3
 #################################
 particles.species_names = electrons ions
 
-warpx.do_nodal = 1
+warpx.grid_type = collocated
 warpx.use_filter = 1
 
 psatd.nox = 16

Examples/Tests/pml/inputs_3d

@@ -15,7 +15,7 @@ boundary.field_hi = pml pml pml
 
 # Numerical parameters
 warpx.cfl = 1.0
-warpx.do_nodal = 0
+warpx.grid_type = staggered
 warpx.do_dive_cleaning = 1
 warpx.do_divb_cleaning = 1
 warpx.do_pml_dive_cleaning = 1

Examples/Tests/vay_deposition/inputs_2d

@@ -59,7 +59,7 @@ psatd.update_with_rho = 0
 
 # warpx
 warpx.cfl = 0.9999
-warpx.do_nodal = 1
+warpx.grid_type = collocated
 warpx.serialize_initial_conditions = 1
 warpx.use_filter = 1
 warpx.verbose = 1

Examples/Tests/vay_deposition/inputs_3d

@@ -61,7 +61,7 @@ psatd.update_with_rho = 0
 
 # warpx
 warpx.cfl = 0.9999
-warpx.do_nodal = 1
+warpx.grid_type = collocated
 warpx.serialize_initial_conditions = 1
 warpx.use_filter = 1
 warpx.verbose = 1

Regression/WarpX-GPU-tests.ini

@@ -367,7 +367,7 @@ numthreads = 1
 compileTest = 0
 doVis = 0
 compareParticles = 0
-runtime_params = warpx.do_dynamic_scheduling=0 warpx.do_nodal=1 algo.current_deposition=direct
+runtime_params = warpx.do_dynamic_scheduling=0 warpx.grid_type=collocated algo.current_deposition=direct
 particleTypes = electrons positrons
 analysisRoutine = Examples/Tests/langmuir/analysis_langmuir_multi.py
 analysisOutputImage = langmuir_multi_analysis.png
@@ -393,7 +393,7 @@ analysisOutputImage = langmuir_multi_analysis.png
 [Langmuir_multi_psatd_nodal]
 buildDir = .
 inputFile = Examples/Tests/langmuir/inputs_3d_multi_rt
-runtime_params = algo.maxwell_solver=psatd warpx.do_dynamic_scheduling=0 warpx.do_nodal=1 algo.current_deposition=direct warpx.cfl = 0.5773502691896258
+runtime_params = algo.maxwell_solver=psatd warpx.do_dynamic_scheduling=0 warpx.grid_type=collocated algo.current_deposition=direct warpx.cfl = 0.5773502691896258
 dim = 3
 addToCompileString = USE_PSATD=TRUE USE_GPU=TRUE
 restartTest = 0
@@ -421,7 +421,7 @@ numthreads = 1
 compileTest = 0
 doVis = 0
 compareParticles = 0
-runtime_params = warpx.do_nodal=1 algo.current_deposition=direct diag1.electrons.variables=w ux uy uz diag1.positrons.variables=w ux uy uz
+runtime_params = warpx.grid_type=collocated algo.current_deposition=direct diag1.electrons.variables=w ux uy uz diag1.positrons.variables=w ux uy uz
 particleTypes = electrons positrons
 analysisRoutine = Examples/Tests/langmuir/analysis_langmuir_multi_2d.py
 analysisOutputImage = langmuir_multi_2d_analysis.png
@@ -447,7 +447,7 @@ analysisOutputImage = langmuir_multi_2d_analysis.png
 # [Langmuir_multi_2d_psatd_nodal]
 # buildDir = .
 # inputFile = Examples/Tests/langmuir/inputs_2d_multi_rt
-# runtime_params = algo.maxwell_solver=psatd warpx.do_nodal=1 algo.current_deposition=direct diag1.electrons.variables=w ux uy uz diag1.positrons.variables=w ux uy uz diag1.fields_to_plot=Ex Ey Ez jx jy jz part_per_cell
+# runtime_params = algo.maxwell_solver=psatd warpx.grid_type=collocated algo.current_deposition=direct diag1.electrons.variables=w ux uy uz diag1.positrons.variables=w ux uy uz diag1.fields_to_plot=Ex Ey Ez jx jy jz part_per_cell
 # dim = 2
 # addToCompileString = USE_PSATD=TRUE USE_GPU=TRUE
 # restartTest = 0