This paper analyzes how a language model processes idioms with non-constructive figurative interpretations through causal tracing. Specifically, we uncover three mechanisms by which a pre-trained causal transformer handles idiom ambiguity: (i) Early lower layers and specific attention heads retrieve figurative interpretations of idioms and suppress literal interpretations. (ii) If context appears before the idiom, the model utilizes it from the earliest layers onward, and if the context conflicts with the retrieved interpretation, it refines the interpretation in later layers. (iii) It propagates both interpretations through selective and competing paths, with the intermediate path prioritizing the figurative interpretation, and the parallel direct path favoring the literal interpretation, ensuring that both interpretations are preserved.
