This paper presents a novel framework that unifies the commonalities of various deep neural network architectures (convolutional, recurrent, and self-attention) into sparse matrix multiplication. Convolution is represented as a first-order transformation via upper triangular matrices, recurrent is represented as a stepwise update via lower triangular matrices, and self-attention is represented as a third-order tensor decomposition, respectively. The authors prove algebraic isomorphism with standard CNN, RNN, and Transformer layers under weak assumptions, and show experimental results on image classification, time series prediction, and language modeling/classification tasks that the sparse matrix formulations match or outperform existing models, converging in a similar or fewer number of epochs. This approach simplifies the architecture design into a sparse pattern selection, which enables GPU parallelization and the utilization of existing algebraic optimization tools.
