This paper presents an integrated framework based on a single Vision-Language Model (VLM) to address the fragmentation and inefficiency of the medical image analysis pipeline. This framework leverages the VLM in two roles. First, the VLM acts as a model card matcher, routing medical images to appropriate specialized models. It performs a three-step process (modality -> major anomaly -> model card ID), with early termination checks at each step improving accuracy. Second, the VLM is fine-tuned on domain-specific datasets to handle multiple subtasks with a single model. In gastroenterology, hematology, ophthalmology, and pathology, single-model deployments demonstrate performance equivalent to or similar to specialized baseline models. This is expected to reduce data scientist effort, accelerate monitoring, increase transparency in model selection, and reduce integration overhead.
