In this paper, we propose an action recognition and control method that uses two long short-term memory (LSTM) deep neural networks and three fully-connected layers to solve complex command (action) classification problems in human-robot interaction (HRI). In particular, we apply federated learning (FL) to enable distributed learning in a multi-robot scenario consisting of multiple drones, thereby performing learning without access to the cloud or other storage. The experimental results using real robots show that the accuracy is over 96%, which solves the problem of human operator occlusion that may occur when operating drones.