This paper focuses on composite AI systems (e.g., agent systems) where multiple large-scale language models (LLMs) specialized for different users, tasks, and roles work together. In such systems, multiple models often process inputs that share the same contextual prefixes. While previous research has focused on prefix KV cache reuse within a single model, prefix KV cache reuse across different models remains an open challenge. In this paper, we present DroidSpeak, the first distributed LLM inference system that enables KV cache reuse across distributed nodes performing inference on different LLMs. For LLMs with the same architecture, DroidSpeak improves inference performance without loss of quality by recomputing only some layers of the KV cache generated by other LLMs and reusing the rest. Additional performance gains are achieved through careful pipelining of layer-wise recomputations and reused KV cache loading. Experimental results on various datasets and model pairs show that DroidSpeak improves throughput by up to 4x and infill time by about 3.1x, with negligible quality loss in F1, Rouge-L, or code similarity scores.
