Incremental Class Learning Approach and Its Application to Handwritten Digit Recognition

Incremental Class Learning (ICL) provides a feasible framework for the development of scalable learning systems. Instead of learning a complex problem at once, ICL focuses on learning subproblems incrementally, one at a time - using the results of prior learning for subsequent learning - and then combining the solutions in an appropriate manner. With respect to multi-class classification problems, the ICL approach presented in this paper can be summarized as follows. Initially the system focuses on one category. After it learns this category, it tries to identify a compact subset of features (nodes) in the hidden layers, that are crucial for the recognition of this category. The system then freezes these crucial nodes (features) by fixing their incoming weights. As a result, these features cannot be obliterated in subsequent learning. These frozen features are available during subsequent learning and can serve as parts of weight structures build to recognize other categories. As more categories are learned, the set of features gradually stabilizes and learning a new category requires less effort. Eventually, learning a new category may only involve combining existing features in an appropriate manner. The approach promotes the sharing of learned features among a number of categories and also alleviates the well-known catastrophic interference problem. We present results of applying the ICL approach to the Handwritten Digit Recognition problem, based on a spatio-temporal representation of patterns.

ICSI Technical Report TR-98-015

Technical Report