This paper presents a novel algorithm and dataset to solve the difficulties of weather heat map interpretation. Existing Vision-Language Models (VLMs) have difficulty in accurately interpreting weather heat maps with irregular outlines and complex color variations. In response, we propose a new algorithm, Sparse Position and Outline Tracking (SPOT), to handle irregularly shaped color regions. SPOT extracts spatial coordinates to represent irregular shapes in a structured manner. Based on SPOT, we construct a new weather visual question-answering (VQA) dataset, ClimateIQA, which consists of 26,280 high-resolution heat maps and 762,120 question-answer samples for analyzing wind speed, precipitation, perceived temperature, and discomfort index. ClimateIQA improves the accuracy of weather heat map interpretation by integrating spatial information, geographical metadata, and reanalysis data. Finally, we develop Climate-Zoo, a collection of fine-tuned VLMs utilizing SPOT and ClimateIQA, which achieves better performance than existing models.