This paper demonstrates that competitive or superior performance can be achieved in a zero-shot approach for the Referring Expression Comprehension (REC) task, without REC-specific training, instead of using task-specific ground truth models. We reconstruct REC using box-wise visual-language verification, where a general-purpose visual language model (VLM) independently answers true/false queries for proposed boxes in the COCO-clean generic detector (YOLO-World). This simple procedure reduces inter-box interference, supports abstention and multiple matches, and requires no fine-tuning. It outperforms the zero-shot GroundingDINO baseline on the RefCOCO, RefCOCO+, and RefCOCOg datasets, as well as reported results from GroundingDINO and GroundingDINO+CRG, which are trained on REC. Control studies using the same proposal demonstrate that verification significantly outperforms choice-based prompting, and the results hold for open-source VLMs. In conclusion, we demonstrate that workflow design, rather than task-specific pretraining, determines robust zero-shot REC performance.
