This paper explores principled approximation methods for improving the efficiency of deep learning systems, focusing particularly on settings involving discrete constraints and non-differentiability. For model compression, we propose novel approximations to pruning and quantization, rendering the underlying discrete problem continuous and differentiable. This approach enables gradient-based learning with compressed model parameters. This approach yields highly compact models without the need for fine-tuning. In terms of architecture design, we design an algorithm that efficiently explores implicitly recursive architectures by leveraging inter-layer parameter sharing. Finally, we study adaptive optimization, revisiting the theoretical properties of widely used methods and proposing an adaptive optimizer capable of rapid hyperparameter tuning. Experimental results on image classification, language modeling, and generative modeling tasks demonstrate that the proposed method significantly improves training and inference efficiency while maintaining or improving model performance.
