In this paper, we propose PVChat, a personalized video large-scale language model (ViLLM). The existing ViLLM has limitations in understanding specific individuals, such as "Wilson is undergoing chemotherapy", while PVChat is designed to enable question-answering (QA) for a specific individual with only a single video. It uses a method of training ViLLM with mixed head (MoH) enhancement on a synthetically extended video-QA dataset. To this end, we introduce an automatic augmentation pipeline that synthesizes positive samples that maintain personal identification information and retrieves difficult speech samples from existing video data, generating various types of QA data such as presence, appearance, action, and location questions. In addition, we propose a ReLU-routed MoH attention mechanism and two new objective functions (Smooth Proximity Regularization, Head Activation Enhancement) to enhance personal feature learning. It enables incremental learning from static attributes to dynamic representations through a two-stage learning strategy that proceeds from image pre-training to video fine-tuning. It outperforms the existing state-of-the-art ViLLM on various datasets such as medical scenarios, TV series, animations, and real-world videos.
