In this paper, we propose a speculative decoding (SPD) method to accelerate the autoregressive token generation process of large-scale language models (LLMs). Existing SPD methods use draft models with multiple heads to predict future token sequences, but they have limitations in that they treat all tokens equally and rely on a single generation method (serial or parallel). In this paper, we theoretically prove that early tokens are more important than later tokens, and based on this, we propose Gumiho, a hybrid model that combines serial and parallel heads. Gumiho uses serial heads with sophisticated Transformer architecture to improve the accuracy of early tokens, and multiple lightweight MLP heads that operate in parallel to improve efficiency. We achieve overall performance improvement by assigning more advanced model structures and longer execution times to early heads. Experimental results show that the proposed method outperforms existing methods.
