In this paper, we propose a novel technique for extracting sentence embeddings from large-scale language models (LLMs), Token Prepending (TP). Existing methods induce LLMs to encode sentence information in the embeddings of the last tokens through prompt engineering, but this leads to biased encoding and cascading effects due to causal attention, which prevents early tokens from referring to later tokens. TP prepends the decoded sentence embeddings of each layer to the input sentences of the next layer, so that early tokens can pay attention to the entire sentence information. It is a plug-and-play, training-free technique that can be seamlessly integrated with various prompt-based sentence embedding methods and autoregressive LLMs. Through extensive experiments on various semantic text similarity (STS) tasks and subclassification tasks, we demonstrate that TP significantly improves the performance of existing methods while barely increasing the inference cost.
