Conjectures on Representations in Backpropagation Networks

The pros and cons of the backpropagation learning procedure have been the subject of numerous debates recently. Some point out its promise as a powerful instrument for finding the weights in a connectionist network appropriate to a given problem, and the generalizability of the solution to novel patterns. Others claim that it is an algorithm for fitting data to a function by error correction through gradient descent. The arguments in this paper focus on the latter (curve-fitting) point of view, but take the point of view that the power of back propagation comes from carefully choosing the form of the function to be fit. This amounts to choosing the architecture and the activation functions of the units (nodes) in the net. A discussion of the role of these two network features motivates two conjectures identifying the form of the squashing function as an important factor in the process. Some preliminary simulations in support of these conjectures are presented.

ICSI Technical Report TR-89-035

Technical Report