Existing generative models for time series forecasting transform simple prior distributions (typically Gaussian) into complex data distributions. However, due to sampling initialization independent of past data, they fail to properly capture temporal dependencies and limit forecast accuracy. Furthermore, they treat residuals merely as optimization objectives, ignoring meaningful patterns such as systematic biases or important distributional structures. In this paper, we propose Conditional Guided Flow Matching (CGFM), a novel model-independent framework that extends flow matching by incorporating the outputs of auxiliary forecasting models. This framework learns from the probabilistic structure of forecast residuals and leverages the predictive distributions of auxiliary models to reduce learning difficulty and improve forecasts. CGFM incorporates historical data as both a condition and a guide, employs bidirectional conditional paths (conditioning on the same past data for both source and target), and employs affine paths to avoid path intersections, maintain temporal consistency, and enhance distributional alignment without complex mechanisms. Experimental results on various datasets and baseline models demonstrate that CGFM consistently outperforms state-of-the-art models, improving forecast performance.