Train a simple neural network using non-linear least squares algorithms (Levenberg-Marquardt), and test them to approximate certain non-linear functions.

A neural network is widely used parametric model to approximate non-linear functions. These networks are parameterized by a set of weights. Neural networks are often referred to as “universal approximators", which means that they can approximate a large class of non-linear functions by suitably tuning the weights. Here I programmed a simple neural network to approximate different types of non-linear functions using “non-linear" least squares algorithms.

A neural network is a model of the form

I used sigmoid as our psi function.

The non-linear function to approximate is

and

I used Levenberg-Marquardt algorithm to determine the weighhts for approximating different non-linear functions.

The loss function is defined as

where r is simply the difference between our prediction fw, and the ground-truth values y.