Project : Ensemble Systems with Negative Correlation Learning

Ensemble Systems are groups of predictors constructed to obtain better generalisation than any single predictor, and have emerged as one of the most powerful pattern recognition techniques of the last decade.
In constructing the base set of learners, it is well appreciated that the individuals should exhibit different patterns of generalisation, the intuition being that a group of identical predictors is no better than a single one of the same form. The performance of the system is dependent these differences, the so called error diversity. A number of techniques in the literature have attempted to encourage diversity by randomly perturbing the training data---Bagging and Random Subspaces are examples of this. In contrast, our work has attempted to explicitly quantify diversity, and incorporate it into a learning algorithm - this is Negative Correlation Learning.

NC learning works with a penalty term attached to the normal MSE error function. A coefficient can be used to vary the emphasis on the penalty term. With a coefficient of zero, the NC learning algorithm is exactly equivalent to a simple ensemble of learners. With a higher coefficient, we have observed significantly faster convergenece, and lower generalisation error on a number of problems.

Demo code:

Download some demonstration code of the algorithm here.

Publications:

Managing Diversity in Regression Ensembles
Gavin Brown, Jeremy Wyatt and Peter Tino
Journal of Machine Learning Research. Volume 6, pp 1621-1650 (2006)
Diversity Creation Methods: A Survey and Categorisation
Gavin Brown, Jeremy Wyatt, Rachel Harris, Xin Yao
Journal of Information Fusion (Special issue on Diversity in Multiple Classifier Systems). Volume 6, issue 1, pp 5-20, March 2005
Examining Decompositions of the Ensemble Objective Function
Gavin Brown, Jeremy Wyatt and Ping Sun
International Workshop on Multiple Classifier Systems. LNCS, Volume 3541, June 2005
Ensemble Learning in Linearly Combined Classifiers via Negative Correlation
Manuela Zanda and Gavin Brown and Giorgio Fumera and Fabio Roli
International Workshop on Multiple Classifier Systems. Prague, May 2007
Diversity in Neural Network Ensembles
Gavin Brown
PhD thesis. University of Birmingham 2004
Negative Correlation Learning and The Ambiguity Family of Ensemble Methods
Gavin Brown and Jeremy Wyatt
International Workshop on Multiple Classifier Systems (MCS'03). Washington DC, USA, August 2003
On The Effectiveness of Negative Correlation Learning
Gavin Brown and Xin Yao
First UK Workshop on Computational Intelligence (UKCI`01). Edinburgh, Scotland, September 2001