Learning Laws for Deep Convolutional Neural Networks With Guaranteed Convergence

Convolutional neural networks (CNNs) have shown remarkable success across numerous tasks such as image classification, yet the theoretical understanding of their convergence remains underdeveloped compared to their empirical achievements. In this paper, the first filter learning framework with convergence-guaranteed learning laws for end-to-end learning of deep CNNs is proposed. Novel update laws with convergence analysis are formulated based on the mathematical representation of each layer in convolutional neural networks. The proposed learning laws enable concurrent updates of weights across all layers of the deep convolutional neural network and the analysis shows that the training errors converge to certain bounds which are dependent on the approximation errors. Case studies are conducted on benchmark datasets and the results show that the proposed concurrent filter learning framework guarantees the convergence and offers more consistent and reliable results during training with a tradeoff in performance compared to stochastic gradient descent methods. This framework represents a significant step towards enhancing the reliability and effectiveness of deep convolutional neural network by developing a theoretical analysis which allows practical implementation of the learning laws with automatic tuning of the learning rate to guarantee the convergence during training.