This paper highlights that existing research on bias in language models (LMs) has primarily focused on data quality, with relatively little attention paid to model architecture and the temporal influence of data. More importantly, few studies have systematically investigated the origins of bias. This study proposes a methodology based on comparative behavioral theory to decipher the complex interactions between training data and model architecture in bias propagation in language modeling. Building on recent research relating Transformers to n-gram LMs, we assess the impact of data, model design choices, and temporal dynamics on bias propagation. Our findings reveal that (1) n-gram LMs are highly sensitive to context window size in bias propagation, whereas Transformers exhibit architectural robustness; (2) the temporal origins of training data significantly influence bias; and (3) different model architectures respond differently to controlled bias injection, with certain biases (e.g., sexual orientation) being disproportionately amplified. Given the widespread use of language models, our findings highlight the need for a holistic approach to mitigating bias, tracing its origins from both the data and model dimensions, rather than simply the symptoms.
