This paper proposes a method for explaining deep learning models for clinical applications in medical image analysis systems. We note that existing techniques, such as GradCAM, can identify influential features but fail to provide explanations themselves. Therefore, we propose a human-machine-Vision-Language Model (VLM) interaction system specifically for explaining classifiers in histopathology. It involves multi-instance learning on entire slide images and quantitatively evaluates the predictive power of explanations using an AI-integrated slide viewer and a standard VLM. Experimental results demonstrate that the proposed system can qualitatively verify explanation claims and quantitatively distinguish competing explanations. This presents a practical approach for advancing explainable AI to explained AI in digital pathology and beyond.
