Artificial Neural Networks

By: Koné Mamadou Tadiou, PhD

A.      Nature of Artificial Neural Network

A Neural Network (NN) is a system composed of several artificial neurons (see Fig.1) and weighted links binding them. This set of neurons that process information, is organized into interconnected layers along chosen patterns. Every neuron in its layer, receives some type of stimuli as input, processes it and sends through its related links an output to neighboring neurons. In the field of Artificial Intelligence [1]  , NN models are also called Artificial Neural Network (ANN) [2]  in contrast to their biological counterparts that strongly inspired researchers in the beginning. There are several kinds of Artificial Neural Network structures according to their topologies and search algorithms. Adapting to new environments is so important for these networks that they rely heavily on learning algorithms to fulfill this need. A typical neural network is an adaptive system made of four main sections:

  • A node as a unit that activates upon receiving incoming signals (inputs);
  • Interconnections between nodes;
  • An activation function (rule) which transforms inside a node,  input into output;
  • An optional learning function for managing weights of input-output pairs.

Artificial Intelligence Neural Network Nodes

Fig. 1: Basic structure of an Artificial Neural Network (ANN)

Current Artificial Neural Network structures have moved from their natural biological model to one that rests more on statistics and signal processing.

B.      Background of Artificial Neural Networks

Neurology inspired McCulloch and Pitts around 1943 to develop models of neural networks. Frank Rosenblatt, a neuro-biologist (Cornell U.) proposed in 1960 his influential Perceptron, a type of computer that could simulate human thinking process and learn under supervision. It was later criticized and dismantled by Marvin Minsky of the MIT. In addition to the influence of neuroscience in the development of neural networks, psychologists and engineers also brought their contribution to their progress with numerous simulations. Following a difficult period triggered by strong criticisms, research on Artificial Neural Network re-emerged around the late 70’s with wider media and academic support through news reports and conferences. For example, in 1987, the Institute of Electrical and Electronic Engineer’s (IEEE) launched the first very popular International Conference on Neural Networks.

C.      Architectures and Methods of Artificial Neural Networks

To deal with abstract and poorly defined problems (face recognition, speech understand-ding, handwriting recognition [5] ), Artificial Neural Network has devised a number of structures, standard methods and techniques. They are the following:

C. 1.  Architectures of neural networks

The most well known classes of Artificial Neural Network are the feed-forward, the feedback and the recurrent neural networks [6]

C. 1.a. Feed-forward and feedback Artificial Neural Network

Feed-forward networks depicted in fig. 1 are characterized by the layout and behaviour of their inner nodes as seen on fig. 2.

 Artificial Intelligence Neural Netowork Internal Node Diagram

Fig. 2: Internal structure of a node with its inputs (xi), weighted inputs (xwi) and Sigmoid function.

  • Their nodes are arranged in input, output and middle layers with the last hidden from the outside;
  • Each node in every layer is connected to every other node in the next layer.
    This guarantees the constant forward flow of information between layers;
  • Nodes within the same layer are totally isolated from one another.

Here the learning process occurs when a sample of input-output pairs is fed into the network repeatedly until it “learns” the relationship between the elements. This process is called supervised learning in feed-forward Artificial Neural Network. In particular, back-propagation learning refers to a supervised learning method where the difference between the output of a learning session and the expected outcome, is as minimal as possible. Perceptrons are a kind of feed-forward Artificial Neural Network used to solve classification and prediction problems.

C. 1.b. The recurrent Artificial Neural Network

A recurrent neural network (RNN) [4], in contrast to its  feed-forward counterpart,  processes any sequence of inputs and organizes its connections between units in the shape of a directed cycle  to display a dynamic temporal behaviour. This ability makes RNN very suitable to tasks such as natural handwriting recognition [5]  in particular and biological recurrent processes in general.

C. 2.  Methods of neural networks

Research in Artificial Neural Network has produced many advanced algorithms in addition to the simple single and multiple layers perceptrons. Some of these algorithms are

  • Kohonen’s unsupervised learning algorithm for two layers network (an input layer and a networked-style layer in a plane) in self-organising networks;
  • Hopfield net  is totally connected, symmetrically weighted with hybrid nodes (input and output);
  • Bumptree network that uses a combination of a binary tree and advanced classification methods using hyper ellipsoids.

D.      Applications of Artificial Neural Network

There are many applications of Artificial Neural Network that surround us. The most popular are in medical science [9] (cancer diagnosis, cardio-vascular system modeling), business (sales forecasting, customer research, target marketing, counterfeit prevention) and manufacturing (resource allocation and scheduling). For example, in marketing, the Airline Marketing Tactician (AMT) rely on Artificial Neural Network-integrated expert system technology to book flights and dynamically allocate seats to customers in an ever changing environment. Some companies have designed Artificial Neural Network-based applications for credit evaluation and mortgage screening also.

Advances in speech recognition, pattern recognition [7]  and image analysis through the implementation of Artificial Neural Network techniques has helped the fields of robotics and intelligent software agents [3]  make significant progress.

NASA launched The Intelligent Flight Control System (IFCS) research with the goal to develop adaptive and fault-tolerant flight control systems and improve aircraft performance in critical conditions.  To achieve this goal, it integrates self-learning neural network concepts into flight control software.

Artificial Intelligence Neural Network Flight Control Systems

E.      Perspectives in Artificial Neural Network

The unusual paradigm, methods and results of research in Artificial Neural Network are so fascinating that  they make us confident that its future is promising and its potential limitless.  First inspired by the human brain and its natural structure, Artificial Neural Network combined with other AI technologies may in return help create one day an artificial or hybrid life-form. With the multitude of commercial applications and research advancing on many fronts, Artificial Neural Network is attracting a lot of attention and also substantial funding.

However, in the modern world, Artificial Neural Network does not fair well with very large and complex problems prevalent in defense, nuclear and space industries. Scalability, instability, verification, and integration are the main issues. The future of Artificial Neural Network could be in their direct implement into the type of hardware still under development as Application Specific Integrated Circuit (ASIC).

F.      References

[1] Russell Stuart. and Norvig Peter. (December 2009), Artificial Intelligence: A Modern Approach, Prentice Hall, ISBN , 1,132 p. - ISBN 0136042597,
URL= http://aima.cs.berkeley.edu/;

[2] Ra´ul Rojas ( 1996). Neural Networks: A Systematic Introduction URL = http://page.mi.fu-berlin.de/rojas/neural/neuron.pdf  ;

[3] Cruse, Holk (2007). Neural Networks as Cybernetic Systems, 2nd ed. URL = http://www.brains-minds-media.org/archive/615/bmm615.pdf ;

[4] Mandic, D. & Chambers, J. (2001). Recurrent Neural Networks for Prediction: Learning Algorithms, Architectures and Stability. Wiley. ISBN 0-471-49517-4;

[5] A. Graves et al. A Novel Connectionist System for Improved Unconstrained Handwriting Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 31, no. 5, 2009;

[6] Weitzenfeld, A. et al. (2002). The Neural Simulation Language: A System for Brain Modeling, The MIT Press;

[7] Bishop, C. M. (1995). Neural Networks for Pattern Recognition, Oxford U. Press;

[8] NASA IFCS  URL =  http://www.nasa.gov/centers/dryden/news/XPress/50th_Artificial Neural Networkiversary/on_the_horizon/ifcs.html ;

[9] Lucila Ohno-Machado (March 1996). PhD dissertation: “Medical Applications of Artificial Neural Networks: Connectionist Models of Survival. URL =  ftp://reports.stanford.edu/pub/cstr/reports/cs/tr/96/1564/CS-TR-96-1564.pdf