This paper introduces Local MAP Sampling (LMAPS), a novel inference framework for solving inverse problems. LMAPS provides a probabilistic interpretation of optimization-based diffusion solvers and iteratively solves local MAP subproblems to follow diffusion trajectories. LMAPS incorporates a probabilistically interpretable covariance approximation, a reconstructed objective function for stability and interpretability, and a gradient approximation for non-differentiable operators. It demonstrates improved performance over existing methods in image restoration and scientific tasks.