This paper presents a novel RGB-only object-level topological navigation pipeline that enables zero-shot, long-range robot navigation without a 3D map or pre-trained controller. It integrates global topological path planning with local metric trajectory control, enabling the robot to navigate to object-level subtargets while avoiding obstacles. It addresses the limitations of existing methods by incorporating a mechanism that continuously predicts local trajectories using monocular depth and traversability estimation and automatically switches to a baseline controller when necessary. Using a baseline model, it ensures open-set applicability without domain-specific fine-tuning. Its robustness and deployability are demonstrated in both simulated and real-world environments, demonstrating its effectiveness and outperforming existing state-of-the-art methods, providing a more adaptable and effective solution for visual navigation in open-set environments. The source code is publicly available.