COGITAO is a modular and extensible data generation framework and benchmark designed to systematically study compositionality and generalization in the vision domain. Inspired by the ARC-AGI problem setting, it constructs rule-based tasks that apply a set of transformations to objects in a grid environment. It supports configurations with adjustable depth for 28 interoperable transformations and offers extensive control over grid parameters and object properties. This flexibility allows the generation of millions of unique task rules (many times more than existing datasets) with varying difficulty and virtually unlimited sample generation per rule. Baseline experiments with state-of-the-art vision models demonstrate that despite excellent domain-specific performance, COGITAO consistently fails to generalize to novel combinations of familiar elements. COGITAO is fully open-source, including all code and datasets, to support ongoing research in this area.
