This paper presents Sequence-Based Structured Causal Modeling (SD-SCM), a framework for specifying causal models with user-defined structures and language model definition mechanisms. We characterize how SD-SCM samples from observation, intervention, and counterfactual distributions according to the desired causal structure. We leverage this procedure to generate individual-level counterfactual data to test popular estimation methods for estimating the mean, conditional mean, and individual treatment effects. We propose a new benchmark for causal inference methods. We create and test an example benchmark consisting of thousands of datasets. We find that (1) causal methods outperform non-causal methods and (2) even state-of-the-art methods struggle with estimating individual effects. This suggests that this benchmark captures the inherent challenges of causal inference. Beyond data generation, this technique can also support the auditing of language models for (undesirable) causal effects, such as misinformation or discrimination. We believe that SD-SCM can serve as a useful tool in any application where continuous data and controllable causal structures are useful.
