A parallel framework for deep learning. Read the paper here.
- Training and inference of dense (fully connected) and convolutional neural networks
- Stochastic gradient descent optimizers: Classic, momentum, Nesterov momentum, and RMSProp
- More than a dozen activation functions and their derivatives
- Loading dense and convolutional models from Keras HDF5 (.h5) files
- Data-based parallelism
Layer type | Constructor name | Supported input layers | Rank of output array | Forward pass | Backward pass |
---|---|---|---|---|---|
Input | input |
n/a | 1, 3 | n/a | n/a |
Dense (fully-connected) | dense |
input1d , flatten |
1 | ✅ | ✅ |
Convolutional (2-d) | conv2d |
input3d , conv2d , maxpool2d , reshape |
3 | ✅ | ❌ |
Max-pooling (2-d) | maxpool2d |
input3d , conv2d , maxpool2d , reshape |
3 | ✅ | ✅ |
Flatten | flatten |
input3d , conv2d , maxpool2d , reshape |
1 | ✅ | ✅ |
Reshape (1-d to 3-d) | reshape |
input1d , dense , flatten |
3 | ✅ | ✅ |
Note: The training of convolutional layers has been discovered to be broken as of release 0.13.0. This will be fixed in a future (hopefully next) release.
Get the code:
git clone https://github.com/modern-fortran/neural-fortran
cd neural-fortran
Required dependencies are:
- A Fortran compiler
- HDF5 (must be provided by the OS package manager or your own build from source)
- functional-fortran, h5fortran, json-fortran (all handled by neural-fortran's build systems, no need for a manual install)
- fpm or CMake for building the code
Optional dependencies are:
- OpenCoarrays (for parallel execution with GFortran)
- BLAS, MKL, or similar (for offloading
matmul
anddot_product
calls) - curl (for downloading testing and example datasets)
Compilers tested include:
- gfortran-9.4.0
- ifort-2021.4
- ifx-2021.4
With gfortran, the following will create an optimized build of neural-fortran:
fpm build \
--profile release \
--flag "-fno-frontend-optimize -I$HDF5INC -L$HDF5LIB"
HDF5 is now a required dependency, so you have to provide it to fpm.
The above command assumes that the HDF5INC
and HDF5LIB
environment
variables are set to the include and library paths, respectively, of your
HDF5 install.
The -fno-frontend-optimize
disables some optimizations that may be harmful
when building neural-fortran.
If you use GFortran and want to run neural-fortran in parallel,
you must first install OpenCoarrays.
Once installed, use the compiler wrappers caf
and cafrun
to build and execute
in parallel, respectively:
fpm build \
--compiler caf \
--profile release \
--flag "-fno-frontend-optimize -I$HDF5INC -L$HDF5LIB"
fpm test \
--profile release \
--flag "-fno-frontend-optimize -I$HDF5INC -L$HDF5LIB"
For the time being, you need to specify the same compiler flags to fpm test
as you did in fpm build
so that fpm knows it should use the same build
profile.
See Fortran Package Manager for more info on fpm.
mkdir build
cd build
cmake .. -DSERIAL=1
make
Tests and examples will be built in the bin/
directory.
If you use GFortran and want to run neural-fortran in parallel,
you must first install OpenCoarrays.
Once installed, use the compiler wrappers caf
and cafrun
to build and execute
in parallel, respectively:
FC=caf cmake ..
make
cafrun -n 4 bin/mnist # run MNIST example on 4 cores
If you want to build with a different compiler, such as Intel Fortran,
set the HDF5_ROOT
environment variable to the root path of your
Intel HDF5 build, and specify FC
when issuing cmake
:
FC=ifort cmake ..
for a parallel build of neural-fortran, or
FC=ifort cmake .. -DSERIAL=1
for a serial build.
To use an external BLAS or MKL library for matmul
calls,
run cmake like this:
cmake .. -DBLAS=-lblas
where the value of -DBLAS
should point to the desired BLAS implementation,
which has to be available in the linking path.
This option is currently available only with gfortran.
To build with debugging flags enabled, type:
cmake .. -DCMAKE_BUILD_TYPE=debug
Type:
ctest
to run the tests.
The easiest way to get a sense of how to use neural-fortran is to look at examples, in increasing level of complexity:
- simple: Approximating a simple, constant data relationship
- sine: Approximating a sine function
- dense_mnist: Hand-written digit recognition (MNIST dataset) using a dense (fully-connected) network
- cnn_mnist: Training a CNN on the MNIST dataset
- dense_from_keras: Creating a pre-trained dense model from a Keras HDF5 file and running the inference.
- cnn_from_keras: Creating a pre-trained convolutional model from a Keras HDF5 file and running the inference.
- get_set_network_params: Getting and setting hyperparameters of a network.
The examples also show you the extent of the public API that's meant to be
used in applications, i.e. anything from the nf
module.
Examples 3-6 rely on curl to download the needed datasets, so make sure you have it installed on your system. Most Linux OSs have it out of the box. The dataset will be downloaded only the first time you run the example in any given directory.
If you're using Windows OS or don't have curl for any other reason, download mnist.tar.gz directly and unpack in the directory in which you will run the example program.
API documentation can be generated with FORD. Assuming you have FORD installed on your system, run
ford ford.md
from the neural-fortran top-level directory to generate the API documentation in doc/html. Point your browser to doc/html/index.html to read it.
This Contributing guide briefly describes the code organization. It may be useful to read if you want to contribute a new feature to neural-fortran.
Thanks to all open-source contributors to neural-fortran: awvwgk, ggoyman, ivan-pi, jacobwilliams, jvdp1, jvo203, milancurcic, pirpyn, rouson, rweed, Spnetic-5, and scivision.
Development of convolutional networks and Keras HDF5 adapters in neural-fortran was funded by a contract from NASA Goddard Space Flight Center to the University of Miami. Development of optimizers is supported by the Google Summer of Code 2023 project awarded to Fortran-lang.
- Fortran Keras Bridge (FKB) by Jordan Ott provides a Python bridge between old (v0.1.0) neural-fortran style save files and Keras's HDF5 models. As of v0.9.0, neural-fortran implements the full feature set of FKB in pure Fortran, and in addition supports training and inference of convolutional networks.
- rte-rrtmgp-nn by Peter Ukkonen is an implementation based on old (v0.1.0) neural-fortran which optimizes for speed and running on GPUs the memory layout and forward and backward passes of dense layers.
- Inference Engine developed at the Berkeley Lab by the Computer Languages and Systems Software (CLaSS) group.
Neural-fortran has been used successfully in over a dozen published studies. See all papers that cite it here.