This paper presents a system called EFAGen that automatically generates executable functional abstractions (EFAs) for mathematical problems. EFAs are programs that infer abstract procedures from specific mathematical problems and solutions to generate new similar problems, and have been limited to elementary mathematics. EFAGen defines a given problem and solution as a program synthesis problem, and formalizes the properties of a valid EFA through executable unit tests. It searches for candidate programs sampled from LLMs and uses execution feedback from unit tests as a reward signal to find EFA programs that faithfully reflect the given problem and solution class. It trains the LLM to become a better EFA writer. Experimental results show that EFAGen can infer faithful EFAs for mathematical problems of various levels, generate learnable problem variants, find harder and easier problem variants, and can be used for data generation.