To address the problem of limited model scalability due to memory bottlenecks, this paper proposes $\textit{DiffusionBlocks}$, a novel framework that transforms Transformer-based networks into independently trainable blocks. It treats residual connections as dynamic system updates and converts them into updates to the denoising process, allowing each block to be trained independently. By leveraging a score matching objective to train each block one at a time, memory requirements are reduced proportionally to the number of blocks. Experiments with various Transformer architectures (vision, diffusion, autoregressive, recurrent depth, and mask diffusion) demonstrate that $\textit{DiffusionBlocks}$ scales to practical tasks while maintaining performance comparable to end-to-end training.
