Artificial Neural Networks (ANNs) originated from an early paper McCullough and Pitts (1943) concerned with modeling the “all-or-none” activity of nervous activity in a biological system. The authors presented a logical model to describe how axons and synapses work in a nervous system. Although they do not necessarily represent actual biological systems, neural network models evolved from this early work and are widely used today in forecasting, classification, and pattern recognition problems.
In this blog, we describe the basic structure of neural network models and discuss their common applications today.
What Are Neural Networks?
Neural networks are nonlinear statistical models that find predictive relationships between any number of inputs to one or more outputs.
A network of filters (called neurons) operate on the inputs with a series of functions and parameters. The fitting or training of a neural network involves estimating the parameters that best describe the relationship between the inputs and the outputs.
Neural Network Models
The functioning unit of a neural network is called a neuron. Figure 1 shows a single threshold neuron with 3 inputs, 3 weights, and a single output, Y. This neuron accepts a linear weighted sum (Z) of the inputs and outputs a 1, if Z >0, and 0, if Z <= 0. (In keeping with the biological analogy, 0 is the threshold for activation for this neuron.) Neural networks extend the binary {1, 0} threshold function to more general functions of Z, increase the number of nodes, add hidden layers, and more.
Figure 2 shows a neuron with a general activation function and Figure 3 shows a similar diagram for 2 neurons, each with their own activation function giving 2 outputs. It’s easy to imagine adding many more inputs (x1, x2, …), and many more neurons and activation functions, as well as additional “hidden” layers in the framework (Figure 4).
Neural Networks "Learning"
As in any machine learning problem, neural networks “learn” on a training set and are then validated on a test or validation set. A training set is a dataset of examples with known outputs that the model is fit to; that is, it sees all the information in the training dataset and its parameters are adjusted to fit it closely.
A test set is a dataset of examples with known outputs that the model doesn’t “learn” from but tries to predict. Depending on how well it predicts the test data, the model can be evaluated for accuracy. Of course, the purpose of any trained model is to predict the output on new examples for which we don’t know the true output. With a model we have confidence in, we can predict the output for these new examples and use that information to make better decisions.
The main unknown parameters are the weights (and biases) each node applies to the inputs. Given a training dataset and initial values for the parameters, the usual procedure for neural networks sends the data forward through the network, calculates an error measure (e.g., average squared error between the predicted output with the real output), and then refines the weights and biases node per node to obtain a closer fit to the outputs. By “closer” we mean that the error measure is made smaller and smaller; a set of weights that yield a minimum error function is found using an optimization method, such as gradient descent. The iteration repeats until some convergence criteria is satisfied.
Back to topWhat Are Neural Networks Used For?
Neural networks provide a very flexible framework for doing pattern recognition, forecasting, and classification.
- Forecasting is predicting one or more quantitative outcomes from both quantitative and nominal input data. Neural networks adapt to changes in a non-stationary series and can produce reliable forecasts even when the series contains a good deal of noise or when only a short series is available for training.
- Pattern recognition is uncovering patterns, typically spatial or temporal, among a set of variables. Common applications might be speech and text recognition, facial recognition, and natural language processing.
- Data classification problems are problems where the output of interest is a category, a value in a discrete list of values. Think of classifying an email as SPAM or not, a loan application as being high risk for default or not, a newly admitted patient’s risk of stroke or not. Many problems have more than two categories, such as in plant classification problems and image recognition problems.
Neural Networks Examples
Medical imaging, self-driving vehicles, geo-science, and many other areas depend on accurate image classification. What has become the canonical example for image classification, and neural networks specifically, is the MNIST database of handwritten digits 0 to 9. An image of a handwritten digit is a 28 x 28 grid of pixels containing grayscale values (values in 0-1).
If we give the neural network enough examples of handwritten 3’s and tell the model that “these are 3’s” how accurately can it recognize a 3 in future images? Similarly, for all the digits, not just 3’s. Check out this site to see how different neural network models (and other models) have performed on this database.
Further examples and details are also available here: IMSL Neural Networks in C.
Back to topNeural Network Algorithms
In this blog, we provided an overview of the artificial neural network, its origins and applications, and described a few examples. In a future blog, we will discuss recent innovations permitting larger and larger networks for solving classification problems with enormous training datasets, especially in the field of digital image recognition.
Want to try and solve your next problem with neural networks? Click the button below to see how IMSL can help add dependable functionality to your application today.