This paper focuses on Large Language Models (LLMs), codes that have shown promising results in translating natural language tasks into programs for service robots. While fine-tuning small, specialized LLMs is of interest, collecting a dataset of task-program pairs specific to each robot is time-consuming and expensive. While methods like SELF-INSTRUCT and EVOL-INSTRUCT can generate new tasks from a few examples, they cannot provide corresponding programs that properly adhere to physical world and robot constraints using the provided programming interface. Using a simulator is a natural potential solution to verify these constraints, but building a simulation environment capable of handling arbitrary tasks and the required objects and locations is challenging. To address this challenge, this paper proposes ROBO-INSTRUCT. ROBO-INSTRUCT opportunistically infers entity properties during program execution and applies these constraints based on how entities are used in the task program, synthesizing a task-specific simulation environment on the fly. Furthermore, ROBO-INSTRUCT integrates an LLM-assisted postprocessing procedure to improve alignment with the robot program. We demonstrate the effectiveness of ROBO-INSTRUCT on several LLMs, showing that the fine-tuned model outperforms all baseline methods and even matches or surpasses the performance of several larger, proprietary models.
