This paper proposes Feature Distillation for model-heterogeneous Federated Learning (FedFD), a novel method for improving knowledge aggregation in heterogeneous model federated learning (Hetero-FL). Existing Hetero-FL utilizes ensemble distillation techniques to improve the performance of a global model using logit distillation, but suffers from the limitation of not being able to compensate for knowledge biases arising from heterogeneous models. To address this issue, FedFD proposes a feature-based ensemble federated knowledge distillation paradigm that improves knowledge integration across heterogeneous models by aligning feature information through orthogonal projections. The server's global model maintains projection layers for each client model architecture to individually align features, and orthogonal techniques are used to reparameterize the projection layers to mitigate knowledge biases and maximize distilled knowledge. Experimental results demonstrate that FedFD outperforms existing state-of-the-art methods.
