We study whether, when a language model is trained on samples from an unknown language K, it can generate valid strings that have not been trained on, capturing the full richness of the language. We question whether consistent and breadth-based language generation (where the model output converges to all unseen strings in K as the training data grows) is possible, and show that this is impossible for a wide range of language models, including next-token prediction models. We also demonstrate that consistent and breadth-based generation is possible when provided with negative examples (strings outside K).
