Neural Information Processing

King's College London MSc in Information Processing and Neural Networks

Information Processing by Neural Networks

The brain performs sophisticated information processing tasks, using hardware and operation rules which are quite different from the ones on which computers are based. The processors in the brain, the neurons, are rather noisy elements which operate in parallel. They are organised in networks, the structure of which can vary from very regular to almost amorphous, and they communicate signals through a huge number of inter-neuron connections. These connections represent the `program' of a network. By continuously updating the strengths of the connections, the network as a whole can modify and optimise its `program', `learn' from experience and adapt to changing circumstances.

From an engineering point of view neurons are in fact rather poor processors, slow and unreliable. In the brain this is overcome by ensuring that always a very large number of neurons are involved in any task, and by having them operate in parallel, with many connections. This is in sharp contrast to conventional computers, where operations are performed sequentially, so that failure of any part of the chain of operations is usually fatal. Conventional computers execute a detailed specification of orders, requiring the programmer to know exactly which data can be expected and how to respond. Subsequent changes in the actual situation, not foreseen by the programmer, lead to trouble. Neural networks adapt to changing circumstances. Finally, in our brain large numbers of neurons end their careers each day unnoticed. Compare this to what happens if we randomly cut a few wires in our workstation.

tabular91

Roughly speaking, conventional computers are the appropriate tools for performing well-defined and rule-based information processing tasks, in stable and perfectly known environments. Neural information processing systems, on the other hand, are superior to conventional computers in dealing with real-world tasks, such as communication (vision, speech recognition), movement coordination (robotics) and experience-based decision making (classification, prediction), where data are often messy, uncertain or even inconsistent, and where perfect solutions are for all practical purposes non-existent.

One can distinguish three types of motivation for studying neural networks. Biologists, physiologists, psychologists and to some degree also philosophers aim at understanding information processing in real biological nervous tissue. They study models, mathematically and through computer simulations, which are preferably close to what is being observed experimentally, and try to understand the global properties and functioning of brain regions.

Engineers and computer scientists would like to understand the principles behind neural information processing in order to use these for designing adaptive software and artificial information processing systems which can also `learn'. They use highly simplified neuron models, which are again arranged in networks. As their biological counterparts, these artificial systems are not programmed, their inter-neuron connections are not prescribed, but they are `trained'. They gradually `learn' to perform tasks by being presented with examples of what they are supposed to do.

The key question then is to understand the relationships between the network performance for a given type of task, the choice of `learning rule' (the recipe for the modification of the connections) and the network architecture. Secondly, engineers and computer scientists exploit the emerging insight into the way real (biological) neural networks manage to process information efficiently in parallel, by building artificial neural networks in hardware, which also operate in parallel. These systems, in principle, have the potential of being incredibly fast information processing machines.

Finally, it will be clear that, due to their complex structure, the large numbers of elements involved, and their dynamic nature, neural network models exhibit a highly non-trivial and rich behaviour. This is why also theoretical physicists and mathematicians have become involved, challenged as they are by the many fundamental new mathematical problems posed by neural network models.

[Return to MSc in Information Processing and Neural Networks]