Medical image segmentation, especially polyp detection, faces the challenge of data scarcity due to annotation that requires expert knowledge. In this paper, we present the SynDiff framework, which combines text-based synthetic data generation with efficient diffusion-based segmentation. We augment the limited training data with semantically diverse samples by generating clinically realistic synthetic polyps via text-conditional inpainting using a latent diffusion model. Unlike existing diffusion methods that require iterative denoising, we introduce direct latent estimation that enables single-step inference that provides computational speedup of T x . On CVC-ClinicDB, SynDiff achieves 96.0% Dice and 92.9% IoU, while maintaining real-time capabilities suitable for clinical deployment. The framework demonstrates that controlled synthetic augmentation improves segmentation robustness without distribution shift. SynDiff bridges the gap between data-hungry deep learning models and clinical constraints, providing an efficient solution for deployment in resource-constrained healthcare environments.