This paper discusses the sparse autoencoder (SAE), which has recently emerged as a powerful tool for interpreting and tuning the internal representation of large-scale language models (LLMs). Existing SAE analysis methods tend to rely solely on input-side activations without considering the causal influence between the model output and each latent feature. This study is based on two main hypotheses: (1) activated latent features do not contribute equally to the model output, and (2) only latent features with high causal influence are effective in model tuning. To test these hypotheses, this paper proposes the Gradient Sparse Autoencoder (GradSAE), a simple yet effective method that integrates output-side gradient information to identify the most influential latent features.
