In this paper, we propose a novel approach, VFedMH (Vertical Federated Learning for training Multiple Heterogeneous models), to solve the optimization convergence and generalization problems caused by heterogeneous local models among participants in vertically distributed learning (VFL). VFedMH focuses on aggregating local embeddings of each participant's knowledge during the forward propagation process. To protect the local embedding values of participants, we propose a lightweight blinding factor-based embedding protection method. The passive party injects the blinding factor into the local embeddings and transmits them to the active party, and the active party aggregates the local embeddings to obtain global knowledge embeddings, which are then transmitted to the passive party. The passive party performs forward propagation in its own local heterogeneous network using the global embeddings. Since the passive party does not own the sample labels, it cannot compute the local model gradients locally, which is overcome by the active party supporting the computation of local heterogeneous model gradients. Each participant learns its own local model using the heterogeneous model gradients, and the goal is to minimize the loss value of each local heterogeneous model. Extensive experiments demonstrate that VFedMH uses heterogeneous optimization to train multiple heterogeneous models simultaneously and outperforms some state-of-the-art methods.
