In this paper, we propose PVChat, a personalized video large-scale language model (ViLLM). Based on the observation that existing ViLLMs are good at general video understanding but struggle with understanding specific individuals (e.g., “Wilson is undergoing chemotherapy”), we present a framework that enables personalized question answering (QA) with a single video. PVChat optimizes ViLLM with mixed head (MoH) enhancement on synthetically augmented video-QA datasets, utilizing an incremental image-video learning strategy. We synthesize identity-preserving positive samples through a data augmentation pipeline and retrieve difficult negative samples from existing video corpora to generate diverse training datasets. In addition, we propose a ReLU-routed MoH attention mechanism and two new objective functions (Smooth Proximity Regularization and Head Activation Enhancement) to enhance personalized feature learning. We adopt a two-stage training strategy from image pretraining to video fine-tuning, enabling an incremental learning process from static attributes to dynamic representations. We evaluate PVChat on a variety of datasets (medical scenarios, TV series, animations, and real-world videos) and demonstrate its superiority over existing state-of-the-art ViLLM in understanding private features after single-video learning.
