This paper studies recent advances in recurrent sub-quadratic models for improving long-context processing efficiency. We investigate leading long-context models, focusing on the impact of fixed-size recurrent memory on performance. Experimental results show that these models underutilize long-context models even when trained with long contexts. We demonstrate that a chunk-based inference procedure, which identifies and processes only the most relevant portions of the input, mitigates recurrent memory failures and is effective for many long-context tasks. On LongBench, the proposed method improves the performance of Falcon3-Mamba-Inst-7B by 14%, Falcon-Mamba-Inst-7B by 28%, RecurrentGemma-IT-9B by 50%, and RWKV6-Finch-7B by 51%. Remarkably, this simple approach achieves state-of-the-art results on the demanding LongBench v2 benchmark, competing with Transformers of the same size. Furthermore, the fact that a single-chunk strategy provides better performance raises the question of whether recurrent models truly utilize long-range dependencies.
