Updtated installation instructions in manual. Changed cover pic

doc/manual/installation.tex

@@ -1,68 +1,188 @@
-All the underlying installation instructions assume a Linux operating system.
+All the underlying installation instructions assume a Linux operating system. Also standard tools and libraries like CMake, compilers- (C, C++ and Fortran), and MPI libraries are assumed to be pre-installed. Most high-performance computers would have the latest version of these libraries in the default environment. However, in many cases you would have to use \href{http:https://modules.sourceforge.net/}{Environment Modules} to set the correct environment variables for compilers-(C, C++ and Fortran), MPI libraries, and compilation tools like \href{http:https://www.cmake.org/}{CMake}. For example, on one of the high-performance computers we use to develop and test the \dftfe{} code, we use the following commands to set the desired environment variables
+\begin{verbatim}
+$ module load cmake
+$ module load intel/18.0.1
+$ module load openmpi/3.0.0/intel/18.0.1
+\end{verbatim}
+We strongly recommend using the latest stable version of compilers-(C, C++ and Fortran), and MPI libraries (only for \dftfe{} and external libraries which require MPI) available on your high-performance computer. Furthermore, the installations which use \href{http:https://www.cmake.org/}{CMake}, version 2.8.12 or later is required. 
+
 \subsection{Compiling and installing external libraries}
-\dftfe{} is primarily based on the open source finite element library \href{http:https://www.dealii.org/}{deal.ii}, through which external dependencies
-on \href{http:https://p4est.org/}{p4est}, \href{https://www.mcs.anl.gov/petsc/}{PETSc}, \href{http:https://slepc.upv.es/}{SLEPc}, and \href{http:https://www.netlib.org/scalapack/}{ScaLAPACK}. ScaLAPACK is an optional requirement, but strongly recommended for large problem sizes with 10,000 electrons or more. The other required external libraries, which are
-not interfaced via deal.ii are \href{http:https://www.alglib.net/}{ALGLIB}, \href{http:https://www.tddft.org/programs/libxc/}{Libxc}, \href{https://atztogo.github.io/spglib/}{spglib}, and \href{http:https://www.xmlsoft.org/}{Libxml2}. Some of the above libraries (PETSc, SLEPc, ScaLAPACK, and Libxml2) are already installed on most high-performance computers. Below, we give brief installation and/or linking instructions for each of the above libraries.
+\dftfe{} is primarily based on the open source finite element library \href{http:https://www.dealii.org/}{deal.II}, through which external dependencies
+on \href{http:https://p4est.org/}{p4est}, \href{https://www.mcs.anl.gov/petsc/}{PETSc}, \href{http:https://slepc.upv.es/}{SLEPc}, and \href{http:https://www.netlib.org/scalapack/}{ScaLAPACK}. ScaLAPACK is an optional requirement, but strongly recommended for large problem sizes with 5000 electrons or more. The other required external libraries, which are
+not interfaced via deal.II are \href{http:https://www.alglib.net/}{ALGLIB}, \href{http:https://www.tddft.org/programs/libxc/}{Libxc}, \href{https://atztogo.github.io/spglib/}{spglib}, and \href{http:https://www.xmlsoft.org/}{Libxml2}. Some of the above libraries (PETSc, SLEPc, ScaLAPACK, and Libxml2) are already installed on most high-performance computers. Below, we give brief installation and/or linking instructions for each of the above libraries.
 
-\subsubsection{Instructions for deal.ii and its dependencies p4est, PETSc, SLEPc, and ScaLAPACK}
-First, the installation instructions for the dependencies:
+\subsubsection{Instructions for deal.II's dependencies p4est, PETSc, SLEPc, and ScaLAPACK}
+First, the installation instructions for the dependencies of deal.II:
 \begin{enumerate}
- \item {\bf p4est}: This library is used by deal.ii to distribute create and distribute finite-element meshes across multiple processors. Download the latest release tarball of p4est from \url{http:https://www.p4est.org/}. 
+ \item {\bf p4est}: This library is used by deal.II to distribute create and distribute finite-element meshes across multiple processors. Download the latest release tarball of p4est from \url{http:https://www.p4est.org/}, and follow the installation instructions in \url{https://www.dealii.org/9.0.0/external-libs/p4est.html}. 
+
+ \item {\bf PETSc}: PETSc is a parallel linear algebra library. \dftfe{} needs two variants of the PETSc installation- one with real scalar type and the another with complex scalar type. Also 64-bit indices must be enabled in both the installation variants. To install PETSc, first download the latest release tarball from \url{https://www.mcs.anl.gov/petsc/download/index.html}. Next unpack using
+\begin{verbatim}
+$ tar -zxvf tarball.tar.gz
+\end{verbatim} 
+, enter into the directory and follow the installation instructions in \url{http:https://slepc.upv.es/documentation/instal.htm}. You have to install two variants of PETSc, a real scalar type with 64b-bit indices installation configured (do \verb|./configure --help| for more information about configuration flags) using
+\begin{verbatim}
+$ ./configure --with-debugging=no --with-64-bit-indices=true
+ [other flags- see below for example]
+\end{verbatim}
+and a complex scalar type with 64-bit indices installation configured using
+\begin{verbatim}
+$ ./configure --with-debugging=no --with-64-bit-indices=true 
+ --with-scalar-type=complex 
+ [other flags- see below for example]
+\end{verbatim}
+After the configuration step, follow the prompts from the terminal output to compile and install PETSc (see example below).
+
+Below, we show an example installation for the real scalar type variant. 
+This example should be used only as a reference.
+\begin{verbatim}
+$ ./configure --prefix=petsc_installation_dir_path --with-debugging=no 
+ --with-64-bit-indices=true --with-cc=c_compiler
+ --with-cxx=c++_compiler --with-fc=fortran_compiler
+ --with-blas-lapack-lib=(optimized BLAS-LAPACK library path) 
+ CFLAGS=c_compilter_flags CXXFLAGS=c++_compiler_flags
+ FFLAGS=fortran_compiler_flags
+
+$ make PETSC_DIR=whatever prompted by PETSc 
+ PETSC_ARCH=whatever prompted by PETSc
+
+$ make PETSC_DIR=whatever prompted by PETSc 
+ PETSC_ARCH=whatever prompted by PETSc
+ install
+\end{verbatim}
+ Please notice that we have used place holders for values of some of the above configuration flags. You have to use the correct values specific to the compilers and MPI libraries you are working with, and based on compiling recommendations for the high-performance computer you are working with. For example, if using Intel compilers and Intel MKL for blas-lapack, it is {\bf very important} to use \href{https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor}{Intel MKL Link Line Advisor} to set the appropriate path for \verb|--with-blas-lapack-lib|, something like
+\begin{verbatim}
+$ --with-blas-lapack-lib="-Wl,--start-group 
+ ${MKLROOT}/lib/intel64/libmkl_intel_lp64.a 
+ ${MKLROOT}/lib/intel64/libmkl_intel_thread.a 
+ ${MKLROOT}/lib/intel64/libmkl_core.a -Wl,--end-group
+ -liomp5 -lpthread -lm -ldl" 
+\end{verbatim}
+ To exploit performance benifit from threads, we recommend (strongly recommended for the new Intel Xeon Phi processors codenamed Knights Landing) linking to threaded versions of Intel MKL libraries by using the options ``threading layer'' and ``OpenMP library'' in \href{https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor}{Intel MKL Link Line Advisor}. The above recommendations for Intel MKL also hold true for ScaLAPACK and deal.II installations below. 
+Finally, for more detalied installation instructions for PETSc see \url{http:https://slepc.upv.es/documentation/instal.htm}.
 
-\item {\bf PETSc}: 
+\item {\bf SLEPc}: The SLEPc library is built on top of PETSc, and it is used for solution of large scale sparse eigenvalue problems on parallel computers. To install SLEPc, first download the latest release tarball from \url{http:https://slepc.upv.es/download/}, and then follow the installation procedure described in \url{http:https://slepc.upv.es/documentation/instal.htm}. {\bf Important: } SLEPc installation requires PETSc to be installed first. You also need to create two separate SLEPc installations- one for PETSc installed with \verb|--with-scalar-type=real|, and the second for PETSc installed with \verb|--with-scalar-type=complex|.
 
-\item {\bf SLEPc}:
+\item {\bf ScaLAPACK}: ScaLAPACK library is used by DFT-FE via deal.II for its parallel linear algebra routines for solving dense linear systems. Installation of ScaLAPACK will not be required in most cases, as most high-performance computers would have ScaLAPACK already installed in the system. For example, in case of Intel MKL the paths for ScaLAPACK and BLACS (ScaLAPACK requires this) libraries would be something like (obtained via \href{https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor}{Intel MKL Link Line Advisor})
+\begin{verbatim}
+${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.so
+${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.so
+\end{verbatim}
+where \verb|$MKLROOT| points to the directory path for Intel MKL library. The above example is shown for Intel MPI library. For Open MPI library, the BLACS path would become something like
+\begin{verbatim}
+${MKLROOT}/lib/intel64/libmkl_blacs_openmpi_lp64.so
+\end{verbatim}
 
-\item {\bf ScaLAPACK}: Installation of ScaLAPACK will not be required in most cases, as most high-performance computers would have ScaLAPACK already installed in the system. For example 
+If you need to install ScaLAPACK library, download the latest release version from \url{http:https://www.netlib.org/scalapack/#\_software}, and build a shared library (use \verb|BUILD_SHARED_LIBS=ON| and \verb|BUILD_STATIC_LIBS=OFF| during the \verb|cmake| configuration) installation of ScaLAPACK using \verb|cmake|. BLACS library, which is required for linking to Intel MKL ScaLAPACK, is not required to be installed separately as it is compiled along with the ScaLAPACK library. Hence you just have to link to \verb|/your_scalapack_installation_dir/lib/libscalapack.so| while compiling deal.II library. {\bf Important: } for best performance, ScaLAPACK must be linked to optimized BLAS-LAPACK libraries by using \verb|USE_OPTIMIZED_LAPACK_BLAS=ON|, and providing external paths to BLAS-LAPACK during the \verb|cmake| configuration.  
 \end{enumerate}
 
-Assuming the above dependencies are installed, we now briefly discuss the installation instructions for the deal.ii library.
+\subsubsection{Instructions for deal.II}
+Assuming the above dependencies are installed, we now briefly discuss the steps to compile and install the deal.II library linked with the above dependencies:
+\begin{enumerate}
+
+\item Obtain the latest release version of deal.II by downloading and unpacking the .tar.gz file from \url{https://www.dealii.org/download.html}. Alternatively (recommended as certain performance enhancements specific to \dftfe{} are missing in the latest release version), you may obtain the forked development version of deal.II library via
+\begin{verbatim}
+$ git clone -b fixIntelCompilation https://github.com/dftfeDevelopers/dealii.git
+\end{verbatim}
+The above forked version has a minor fix which resolves compilation issues when using intel compilers. \textcolor{red}{Once this fix is available in the deal.II development repository, users can obtain the development version of deal.II library via}
+\begin{verbatim}
+$ git clone https://github.com/dealii/dealii.git
+\end{verbatim}
+
+
+\item
+\begin{verbatim}
+mkdir build
+$ cd build
+$ cmake -DCMAKE_INSTALL_PREFIX=/path/to/install/dir otherCmakeOptions ../deal.II
+$ make install
+\end{verbatim}
+``otherCmakeOptions'' {\bf must include} the following options
+\begin{verbatim}
+-DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_64BIT_INDICES=ON
+-DP4EST_DIR=p4est_install_dir_path
+-DDEAL_II_WITH_PETSC=ON -DDEAL_II_WITH_SLEPC=ON
+\end{verbatim}
+and LAPACK and ScaLAPACK link options. 
+\end{enumerate} 
+Below we provide an example of deal.II installation, which we did on a high-performance computer (\href{https://www.tacc.utexas.edu/systems/stampede2}{STAMPEDE2}) using Intel compilers and Intel MPI library
+\begin{verbatim}
+$ mkdir build
+$ cd build
+$ cmake -DCMAKE_C_COMPILER=mpicc -DCMAKE_CXX_COMPILER=mpicxx 
+ -DCMAKE_Fortran_COMPILER=mpif90 -DDEAL_II_CXX_FLAGS_RELEASE=-xMIC-AVX512 
+ -DDEAL_II_CXX_FLAGS_DEBUG=-xMIC-AVX512 -DDEAL_II_COMPONENT_EXAMPLES=OFF
+ -DDEAL_II_WITH_MPI=ON -DDEAL_II_WITH_64BIT_INDICES=ON
+ -DP4EST_DIR=p4est_install_dir_path
+ -DDEAL_II_WITH_PETSC=ON 
+ -DPETSC_DIR=petsc_install_dir_path
+ -DDEAL_II_WITH_SLEPC=ON
+ -DSLEPC_DIR=petsc_install_dir_path
+ -DDEAL_II_WITH_LAPACK=ON
+ -DLAPACK_DIR="${MKLROOT}/lib/intel64" -DLAPACK_FOUND=true
+ -DLAPACK_LIBRARIES="${MKLROOT}/lib/intel64/libmkl_intel_lp64.so;
+ ${MKLROOT}/lib/intel64/libmkl_core.so;${MKLROOT}/lib/intel64/libmkl_intel_thread.so" 
+ -DLAPACK_LINKER_FLAGS="-liomp5 -lpthread -lm -ldl"
+ -DSCALAPACK_DIR="${MKLROOT}/lib/intel64"
+ -DSCALAPACK_LIBRARIES="${MKLROOT}/lib/intel64/libmkl_scalapack_lp64.so;
+ ${MKLROOT}/lib/intel64/libmkl_blacs_intelmpi_lp64.so"
+ -DCMAKE_INSTALL_PREFIX=dealii_install_dir_path
+ ../dealii
+$ make -j 8
+$ make install
+\end{verbatim}
+The values for \verb|-DLAPACK_DIR|,\verb|-DLAPACK_LIBRARIES|, \verb|-DLAPACK_LINKER_FLAGS|,\verb|-DSCALAPACK_DIR|, and\\ \verb|-DSCALAPACK_LIBRARIES| were obtained with the help of \href{https://software.intel.com/en-us/articles/intel-mkl-link-line-advisor}{Intel MKL Link Line Advisor}. 
 
-It is {\bf very important} to ensure that deal.ii and its dependencies (p4est, PETSc, SLEPc, and ScaLAPACK), and the \dftfe{} source code are compiled with the same compilers and MPI libraries, to prevent occurence of crashes and runtime slowdown. 
+For more information about installing deal.II library refer to \url{https://dealii.org/developer/readme.html}. If you are installing the development version of the deal.II library, sometimes you may run into cross platform compilation issues. If you face such an issue, and/or if you have any questions about the deal.II installation, please contact the deal.II developers at \href{https://groups.google.com/d/forum/dealii}{deal.II mailing lists}.
+
+\textcolor{red}{\bf It is very important to ensure that deal.II and its dependencies (p4est, PETSc, SLEPc, and ScaLAPACK), are compiled with the same compilers, BLAS-LAPACK libraries, and MPI libraries to prevent deal.II compilation issues, occurence of runtime crashes, and \dftfe{} performance degradation.} 
 
 \subsubsection{Instructions for ALGLIB, Libxc, spglib, and Libxml2}
 \begin{enumerate}
  \item {\bf ALGLIB}: Used by \dftfe{} for spline fitting. Download the latest release of the Alglib free C++ edition from \url{http:https://www.alglib.net/download.ph}. After downloading and unpacking, go to \verb|cpp/src|, and create a shared library using a C++ compiler. For example, using g++ compiler do
 \begin{verbatim}
-g++ -c -fPIC *.cpp
-g++ *.o -shared -o libAlglib.so
+$ g++ -c -fPIC *.cpp
+$ g++ *.o -shared -o libAlglib.so
 \end{verbatim}
 \item {\bf Libxc}: Used by \dftfe{} for exchange-correlation functionals. Download the latest release from \url{http:https://www.tddft.org/programs/libxc/download/}, and follow the recommended installation procedure (using \verb|./configure|) described in \url{http:https://www.tddft.org/programs/libxc/installation/}.
 
-\item {\bf spglib}: Used by \dftfe{} to find and handle crystal symmetries. Obtain the source code of the development, and follow the installation procedure described in \url{https://atztogo.github.io/spglib/install.html}. 
+\item {\bf spglib}: Used by \dftfe{} to find crystal symmetries. To install spglib, first obtain the development version of spglib from their github repository by
+\begin{verbatim}
+$ git clone https://github.com/atztogo/spglib.git 
+\end{verbatim} 
+and next follow the ``Compiling using cmake'' installation procedure described in \url{https://atztogo.github.io/spglib/install.html}. 
 
-\item {\bf Libxml2}: Libxml2 is used by \dftfe{} to read \verb|.xml| files. Usually, Libxml2 is already installed on most high-performance computers. It is usually installed in the default locations like \verb|/usr/lib64| (library path) and \verb|/usr/include/libxml2| (include path). Libxml2 can also be installed by doing
+\item {\bf Libxml2}: Libxml2 is used by \dftfe{} to read \verb|.xml| files. Most likely, Libxml2 is already installed in the high-performance computer you are working with. It is usually installed in the default locations like \verb|/usr/lib64| (library path) and \verb|/usr/include/libxml2| (include path). Libxml2 can also be installed by doing
 \begin{verbatim}
-git clone git:https://git.gnome.org/libxml2 
+$ git clone git:https://git.gnome.org/libxml2 
 \end{verbatim}
 and following the installation instructions in the README.
 \end{enumerate}
 
 \subsection{Obtaining and Compiling \dftfe{}}
-Assume that you have already installed the above external dependencies, next follow the steps below to obtain and compile \dftfe{}.
+Assuming that you have already installed the above external dependencies, next follow the steps below to obtain and compile \dftfe{}.
 \begin{enumerate}
 \item Download the source code of the latest release of \dftfe{} from \href{wesbite}{here}. After downloading, unpack the file using the command
 \begin{verbatim}
-tar -xvzf dftfe-0.5.0.tar.gz
+$ tar -zxvf dftfe-0.5.0.tar.gz
+\end{verbatim}
+For obtaining the development version, clone the development branch of our github repository directly on the command line via
+\begin{verbatim}
+$ git clone -b publicGitHubDevelop https://github.com/dftfeDevelopers/dftfe
 \end{verbatim}
 
 \item \begin{verbatim}
-cd dftfe
+$ cd dftfe
 \end{verbatim}
 
 
-\item Make sure the Environment Variables for \href{http:https://www.cmake.org/}{CMake} (version 2.8.12 or later), C and C++ compilers, and MPI libraries are correctly set to the desired choices. In most high-performance computers the paths for these can be set using \href{http:https://modules.sourceforge.net/}{Environment Modules}. For example, on one of the high-performance computers we develop and test the DFT-FE code, we use the following commands to set the environment
-\begin{verbatim}
-module load cmake
-module load intel/18.0.1
-module load openmpi/3.0.0/intel/18.0.1
-\end{verbatim}
-
-\item Set paths to external libraries (deal.ii, ALGLIB, Libxc, spglib, and Libxml2), compiler options, and compiler flags in \verb|setup.sh|, which is a script to compile \dftfe{}. A few example \verb|setup.sh| scripts are provided in the \verb|/helpers| folder for your reference. 
+\item Set paths to external libraries (deal.II, ALGLIB, Libxc, spglib, and Libxml2), compiler options, and compiler flags in \verb|setup.sh|, which is a script to compile \dftfe{}. For your reference, a few example \verb|setup.sh| scripts are provided in the \verb|/helpers| folder. 
+
+Please make sure to set \verb|withIntelMkl=ON| in setup.sh if you have installed deal.II by linking with Intel MKL library, otherwise set it to \verb|OFF|. 
 
 \item Set \verb|optimizedFlag=1| in \verb|setup.sh| and do
 \begin{verbatim}
-./setup.sh
+$ ./setup.sh
 \end{verbatim} 
 which compiles \dftfe{} in the release mode (the fast version). If compilation is successful, a \verb|/build| directory will be created with the following executables:
 \begin{verbatim}
@@ -71,7 +191,7 @@ \subsection{Obtaining and Compiling \dftfe{}}
 \end{verbatim}
 One could also set \verb|optimizedFlag=0| in \verb|setup.sh| and do
 \begin{verbatim}
-./setup.sh
+$ ./setup.sh
 \end{verbatim}
 which will create debug mode executables (useful for debugging purposes but much slower than the release mode executable) in the folders
 \begin{verbatim}