This paper proposes CLAP (Conditional LAtent Point-diffusion model), a novel method for accurate 3D modeling of human organs, such as the colon, which are challenging due to their complex geometry and shape variation. CLAP enhances the 3D representation of the colon by combining geometric deep learning with a denoising diffusion model. Using point clouds sampled from segmentation masks, we learn global and local latent shape representations via hierarchical variational autoencoders, and then refine the organ shape within the latent space using two conditional diffusion models. Finally, we transform the refined point cloud into a mesh using a pretrained surface reconstruction model. Experimental results show that CLAP significantly improves shape modeling accuracy, reducing the Chamfer distance by 26% and the Hausdorff distance by 36% compared to the initial shape. This provides a robust and scalable high-fidelity organ modeling solution applicable to a variety of anatomical structures.