This paper addresses the vulnerability of skeleton-based neural networks for sub-second future human motion prediction, especially to evasion attacks and backdoor attacks. We propose BadHMP, a novel backdoor attack specifically targeting the human motion prediction task. BadHMP uses toxic training samples generated by inserting backdoor triggers localized to a part of the skeleton. These triggers cause certain joints to follow a predefined motion in the past time step, and then the future sequences are globally modified so that all joints move along the target trajectory. The carefully designed backdoor triggers and targets ensure the smoothness and naturalness of the toxic samples, making them difficult for the model trainer to detect, while making the toxic model invisible in terms of prediction fidelity for untainted sequences. The designed input sequences can successfully activate the target sequences even at low toxic sample injection rates. Experimental results on two datasets, Human3.6M and CMU-Mocap, and two network architectures, LTD and HRI, demonstrate the high fidelity, effectiveness, and stealth of BadHMP. The robustness of the attack against fine-tuned defenses has also been verified.