This paper presents a multifaceted approach to solving the problem of wrist lesion recognition, a common problem in pediatric fracture patients, using a limited dataset. First, we approach wrist lesion recognition as a fine-grained image recognition problem and enhance network performance by integrating patient metadata with X-ray images. Furthermore, we further improve performance by leveraging weights learned from a separate, fine-grained image dataset. While metadata integration has been used in other medical fields, this is the first study to apply it to wrist lesion recognition.