In this paper, we present a matrix ordering framework that unifies various deep learning architectures (convolutional, recurrent, and self-attention). We express convolutional, recurrent, and self-attention operations as sparse matrix multiplications, which are implemented as upper triangular matrices, lower triangular matrices, and third-order tensor decompositions, respectively. We prove that the proposed framework is algebraically isomorphic with standard CNN, RNN, and Transformer layers under mild assumptions, and experimentally demonstrate that it performs as well or better than existing models on image classification, time series prediction, and language modeling/classification tasks. We simplify the architectural design to sparse pattern selection, which allows us to leverage GPU parallelism and existing algebraic optimization tools.
