BlobCtrl is a framework for precise manipulation of specific visual elements using a diffusion-based method. By treating blobs as visual primitives, it separates layout and appearance, enabling precise object-level manipulation. Key contributions include: (1) an in-context dual-branch diffusion model that explicitly separates layout and appearance by separating foreground and background processing and unifying blob representations; (2) a self-supervised split-and-reconstruct training paradigm with an identity-preserving loss function, and a tailored strategy for efficiently utilizing blob-image pairs. To facilitate research, we introduce BlobData for large-scale training and BlobBench for systematic evaluation. Experimental results demonstrate that BlobCtrl achieves state-of-the-art performance while maintaining computational efficiency across a variety of element-level editing operations, such as adding, removing, resizing, and replacing objects.