Cyclical Local Structural Risk Minimization with Growing Neural Networks

TitleCyclical Local Structural Risk Minimization with Growing Neural Networks
Publication TypeTechnical Report
Year of Publication1996
AuthorsLange, J. Matti
Other Numbers1025
Abstract

With that paper a new concept for learning from examples called Cyclical Local Structural Risk Minimization (CLSRM) minimizing a global risk by cyclical minimization of residual local risks is introduced. The idea is to increase the capacity of the learning machine cyclically only in those regions where the effective loss is high and to do a stepwise local risk minimization, restricted to those regions. An example for the realization of the CLSRM principle is the TACOMA (TAsk Decomposition, COrrelation Measures and local Attention neurons) learning architecture. The algorithm generates a feed-forward network bottom up by cyclical insertion of cascaded hidden layers. The output of a hidden unit is locally restricted with respect to the network input space using a new kind of activation function combining the local characteristic of radial basis functions with sigmoid functions. The insertion of such hidden units increases the capacity only locally and leads finally to a neural network with a capacity well adapted to the distribution of the training data. The performance of the algorithm is shown for classification and function approximation benchmarks.

URLhttp://www.icsi.berkeley.edu/ftp/global/pub/techreports/1996/tr-96-015.pdf
Bibliographic Notes

ICSI Technical Report TR-96-015

Abbreviated Authors

J. M. Lange

ICSI Publication Type

Technical Report