This paper highlights that the existing parameter-efficient fine-tuning (PEFT) method, LoRA, relies on a global low-dimensional structure, which can overlook spatial patterns. We propose a Localized LoRA framework that models weight updates as a synthesis of low-dimensional matrices applied to structured blocks of the weight matrix. This enables dense, localized updates across the entire parameter space without increasing the total number of learnable parameters. A formal comparison between global, diagonal-local, and fully local low-dimensional approximations demonstrates that the proposed method consistently achieves lower approximation errors under matched parameter budgets. Experiments on synthetic and real-world settings demonstrate that Localized LoRA is a more expressive and adaptive alternative to existing methods, enabling efficient fine-tuning with improved performance.
