This paper proposes a cost-effective framework called Cequel to address the high cost of text clustering using large-scale language models (LLMs). Cequel uses algorithms called EdgeLLM and TriangleLLM to selectively query LLMs for information-rich text pairs or triplets, generating must-link and cannot-link constraints. These constraints are then used in a weighted constraint clustering algorithm to form high-quality clusters. EdgeLLM and TriangleLLM efficiently identify and extract information-rich constraints using a carefully designed greedy selection strategy and prompting technique. Experimental results on various benchmark datasets demonstrate that Cequel outperforms existing unsupervised text clustering methods within the same query budget.