This paper proposes RT-Cache, a training-free search-based control pipeline that overcomes the limitations of existing controllers, which often have high step-by-step inference costs or require fine-tuning during deployment, for real-world robots that must repeat identical actions in novel environments with very little data. RT-Cache caches various image-action paths in a unified vector memory and replaces step-by-step model invocation by embedding the current frame at test time to retrieve and replay multi-step snippets. Hierarchical search maintains sub-second search performance even at scales in the millions, translating computational costs into storage capacity and enabling real-time control on modest GPUs. On real-world robot tasks and large-scale open logs, RT-Cache achieves higher success rates and shorter completion times than robust search baseline models (approximately 2X higher success rates and ~30% faster speedup in our setup). Single-episode fixation studies demonstrate immediate adaptation to more complex, high-touch tasks without fine-tuning. RT-Cache provides a foundation for simple, scalable paths that can be deployed in a few trials, and for optional integration with multi-mode keys and higher-level policies by transferring experience to additional dedicated memory.
