In this paper, we propose a model-agnostic latent space idea generation framework to overcome the limitations of large-scale language models (LLMs) in their creative idea generation capabilities. Unlike existing domain-specific heuristics or structured prompt pipeline approaches, our approach enables controllable and scalable creativity through continuous embedding space exploration that is easily applicable to a variety of domains, input formats, and creative tasks without manual rules. In this paper, we introduce an early prototype of our method and present the conceptual framework and preliminary results demonstrating its potential as a general-purpose joint idea generator for human-AI collaboration.