Interacting with real objects in mixed reality (MR) is often challenging when objects are dense, distant, or partially occluded. This difficulty arises because inputs are tightly coupled to physical constraints, and thus interact directly with physical objects. In this paper, we present a key insight to decouple interactions from these constraints by introducing proxies, abstract representations of real objects. The Reality Proxy system seamlessly transitions the interaction target from the physical object to the proxy during selection. In addition to facilitating basic selection, Reality Proxy uses AI to enrich the proxy with semantic properties and hierarchical spatial relationships of the objects, enabling novel interactions such as skimming, attribute-based filtering, nested group navigation, and complex multi-object selection without the need for a new gesture or menu system. We demonstrate Reality Proxy’s versatility in a variety of scenarios, including office information retrieval, large-scale spatial navigation, and multi-drone control. Expert evaluations demonstrate the utility and usability of the system, suggesting that proxy-based abstractions provide a powerful and generalizable interaction paradigm for future MR systems.
