This paper proposes a PrivacyGuard framework to solve the Privacy-sensitive Object Identification (POI) problem by interpreting POI as a visual reasoning task to determine the privacy class of each object. PrivacyGuard comprises i) a Structuring step that constructs a heterogeneous scene graph rich in scene context, ii) a Data Augmentation step that proposes a contextual perturbation oversampling strategy to address the imbalance of privacy classes, and iii) a Hybrid Graph Generation & Reasoning step that generates a hybrid graph that allows direct message passing between nodes and edges to capture subtle contextual changes.