This paper argues that, unlike humans' ability to learn from single examples, robots struggle with generalization, arguing that this is due to their inability to recover the underlying explanation (latent program) of intelligent behavior. To address this, we propose a Rational Inverse Reasoning (RIR) framework that infers latent programs through a hierarchical generative model of behavior. RIR addresses small-shot imitation through a Bayesian program induction approach, where a vision-language model iteratively proposes structured symbolic task hypotheses, and a planner-based inference system evaluates each hypothesis based on the likelihood of observed exemplars. This process yields a posterior probability for a concise and feasible program. We evaluate RIR on a set of continuous manipulation tasks, assessing single-shot and small-shot generalization across a variety of object poses, counts, geometric shapes, and arrangements. We demonstrate that RIR can infer the intended task structure and generalize to new settings with just a single example, outperforming state-of-the-art vision-language model baselines.
