Deep architectures (DAs) refer to a family of computational models composed of multiple levels of nonlinear operations, e.g., neural networks or Bayesian belief networks of many hidden layers. Researches in neuroscience uncover that the brain extracts multiple levels of representations from sensory input to achieve its impressive performance in various perceptual tasks ranging from visual to speech perception. Motivated by the neuroscience researches, computational learning theories show that modeling complex behaviors, such as machine perception, requires highly varying nonlinear mathematical functions, which expresses the high-level abstraction with respect to sensory input. Furthermore, the recent theoretical outcome also suggests that existing shallow architectures, e.g., SVMs, suffer from theoretical limitations so that they are less competent for efficiently learning highly varying nonlinear functions demanded in modeling complex behaviors. As a result, DAs become an inevitable choice to develop complex intelligent systems. Learning DAs turns out to be a very difficult optimization task; in general, the traditional back-propagation algorithm based solely on supervised learning does not work well in training DAs. Recently, a breakthrough has been made in learning DAs with a hybrid learning framework and the use of DAs trained by hybrid learning algorithms leads to the state-of-the-art performance in several challenging machine perception tasks.
This tutorial would cover main topics in state-of-the-art DAs and their successful applications in a systematic way, which consists of four parts. Within the first part, the background and core concepts of DAs will be introduced and the theoretical justification of DAs will also be briefly discussed. The second part of this tutorial will overview typical building blocks of DAs, e.g., restricted Boltzmann machine (RBM) and de-noising auto-associator (DNAA), basic learning algorithms, e.g., divergence contrastive and stochastic gradient descent algorithms, for building blocks and the essential hybrid learning strategy for learning DAs by combining the local greedy layerwise unsupervised learning and global supervised learning via fine-tuning parameters. The third part will present a number of successful applications of DAs in machine perception, e.g., hand-written OCR recognition, object recognition and learning intrinsic speaker-specific characteristics. The last part will cover the latest progresses in DA researches as well as discuss relevant issues and open research topics.
In summary, this tutorial provides a deep understanding of the problems and the currently proposed solutions to DAs and their applications in real world problems. The tutorial would help postgraduate researchers and practitioners in neural computation, machine learning and pattern recognition gain the knowledge of DAs and the insight into the current state of DAs as well as their applications in real world problems.
Updated: K. Chen 10-Jan-2011