This paper presents a novel method for improving the accuracy and robustness of object pose estimation by integrating visual and haptic information. To address the challenges of previous studies, which include diverse grippers, sensor placements, lack of generalization between simulation and real environments, and inconsistencies in tracking due to frame-by-frame independent estimation, we propose a unified tactile representation that effectively handles multiple gripper implementations and a visual-haptic transformer-based object pose tracker that seamlessly integrates visual and haptic inputs. The proposed method achieves excellent generalization and robustness across diverse implementations, objects, and sensor types (both taxon-based and vision-based tactile sensors), significantly outperforming state-of-the-art visual trackers in real-world experiments. Furthermore, we demonstrate that real-time object tracking can be integrated into motion planning to enable precise manipulation tasks.
