Sign In

SPEED-Bench: A Unified and Diverse Benchmark for Speculative Decoding

์ž‘์„ฑ์ž
  • Haebom
์นดํ…Œ๊ณ ๋ฆฌ
Empty

์ €์ž

Talor Abramovich, Maor Ashkenazi, Izzy Putterman, Benjamin Chislett, Tiyasa Mitra, Bita Darvish Rouhani, Ran Zilberstein, Yonatan Geifman

๐Ÿ’ก ๊ฐœ์š”

๋ณธ ๋…ผ๋ฌธ์€ ๋Œ€๊ทœ๋ชจ ์–ธ์–ด ๋ชจ๋ธ(LLM) ์ถ”๋ก  ๊ฐ€์†ํ™” ๊ธฐ์ˆ ์ธ ์˜ˆ์ธก ๋””์ฝ”๋”ฉ(Speculative Decoding, SD)์˜ ํšจ๊ณผ์ ์ธ ํ‰๊ฐ€๋ฅผ ์œ„ํ•ด SPEED-Bench๋ผ๋Š” ํ†ตํ•ฉ์ ์ด๊ณ  ๋‹ค์–‘ํ•œ ๋ฒค์น˜๋งˆํฌ๋ฅผ ์ œ์•ˆํ•œ๋‹ค. ๊ธฐ์กด ๋ฒค์น˜๋งˆํฌ์˜ ์ œํ•œ๋œ ์ž‘์—… ๋‹ค์–‘์„ฑ, ์ฒ˜๋ฆฌ๋Ÿ‰ ํ‰๊ฐ€ ๋ฏธํก, ์‹ค์ œ ํ™˜๊ฒฝ๊ณผ์˜ ๊ดด๋ฆฌ๋ฅผ ํ•ด๊ฒฐํ•˜๊ณ ์ž, ์˜๋ฏธ๋ก ์  ๋‹ค์–‘์„ฑ์„ ๊ฐ–์ถ˜ ์งˆ์  ๋ฐ์ดํ„ฐ์…‹๊ณผ ๋‹ค์–‘ํ•œ ๋™์‹œ์„ฑ ์ˆ˜์ค€์„ ๊ณ ๋ คํ•œ ์ฒ˜๋ฆฌ๋Ÿ‰ ๋ฐ์ดํ„ฐ์…‹์„ ํฌํ•จํ•œ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ํ”„๋กœ๋•์…˜ ์—”์ง„๊ณผ์˜ ํ†ตํ•ฉ์œผ๋กœ ์‹ค์ œ ์‹œ์Šคํ…œ ๋™์ž‘์„ ๋ถ„์„ํ•˜๊ณ , ํ•ฉ์„ฑ ์ž…๋ ฅ์˜ ์ฒ˜๋ฆฌ๋Ÿ‰ ๊ณผ๋Œ€ํ‰๊ฐ€, ๋ฐฐ์น˜ ํฌ๊ธฐ์— ๋”ฐ๋ฅธ ์ตœ์  ๋“œ๋ž˜ํ”„ํŠธ ๊ธธ์ด, ๋‚ฎ์€ ๋‹ค์–‘์„ฑ ๋ฐ์ดํ„ฐ์˜ ํŽธํ–ฅ ๋“ฑ SD์˜ ๋‹ค์–‘ํ•œ ์ธก๋ฉด์„ ์ •๋Ÿ‰ํ™”ํ•œ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
์˜ˆ์ธก ๋””์ฝ”๋”ฉ(SD) ์„ฑ๋Šฅ์€ ์‹ค์ œ ๋ฐ์ดํ„ฐ ๋ถ„ํฌ์— ๋”ฐ๋ผ ํฌ๊ฒŒ ๋‹ฌ๋ผ์ง€๋ฏ€๋กœ, ๋‹ค์–‘ํ•œ ์‹œ๋งจํ‹ฑ ๋„๋ฉ”์ธ๊ณผ ์‹ค์ œ ์„œ๋น„์Šค ํ™˜๊ฒฝ์„ ๋ฐ˜์˜ํ•˜๋Š” ๋ฒค์น˜๋งˆํฌ์˜ ์ค‘์š”์„ฑ์ด ๊ฐ•์กฐ๋œ๋‹ค.
โ€ข
SPEED-Bench๋Š” ๋‹ค์–‘ํ•œ ๋™์‹œ์„ฑ ์ˆ˜์ค€(์ €๋ฐฐ์น˜๋ถ€ํ„ฐ ๊ณ ๋ถ€ํ•˜๊นŒ์ง€)์—์„œ์˜ ์ฒ˜๋ฆฌ๋Ÿ‰ ํ‰๊ฐ€๋ฅผ ์ง€์›ํ•˜์—ฌ, ์‹ค์ œ ์„œ๋น„์Šค ํ™˜๊ฒฝ์—์„œ์˜ SD ์„ฑ๋Šฅ์„ ๋ณด๋‹ค ์ •ํ™•ํ•˜๊ฒŒ ์ธก์ •ํ•  ์ˆ˜ ์žˆ๋Š” ๊ธฐ๋ฐ˜์„ ์ œ๊ณตํ•œ๋‹ค.
โ€ข
ํ•ฉ์„ฑ ๋ฐ์ดํ„ฐ๋ฅผ ์‚ฌ์šฉํ•œ ํ‰๊ฐ€๋Š” ์‹ค์ œ ์ฒ˜๋ฆฌ๋Ÿ‰๊ณผ ์ฐจ์ด๊ฐ€ ํด ์ˆ˜ ์žˆ์œผ๋ฉฐ, ๋ฐ์ดํ„ฐ ๋‹ค์–‘์„ฑ์ด ๋‚ฎ์„ ๊ฒฝ์šฐ ํŠน์ • ๋“œ๋ž˜ํ”„ํŠธ ๊ธธ์ด ์„ค์ •์ด๋‚˜ ์–ดํœ˜ ๊ฐ€์ง€์น˜๊ธฐ(vocabulary pruning)์™€ ๊ฐ™์€ ๊ธฐ๋ฒ•์— ํŽธํ–ฅ๋œ ๊ฒฐ๊ณผ๊ฐ€ ๋‚˜์˜ฌ ์ˆ˜ ์žˆ์Œ์„ ์‹œ์‚ฌํ•œ๋‹ค.
โ€ข
ํ–ฅํ›„ ๋” ๋„“์€ ๋ฒ”์œ„์˜ LLM ๋ชจ๋ธ๊ณผ ๋‹ค์–‘ํ•œ ํ•˜๋“œ์›จ์–ด ํ™˜๊ฒฝ์—์„œ์˜ ํ‰๊ฐ€ ํ™•์žฅ ๋ฐ ์ง€์†์ ์ธ ๋ฐ์ดํ„ฐ์…‹ ์—…๋ฐ์ดํŠธ๊ฐ€ ํ•„์š”ํ•˜๋‹ค.
๐Ÿ‘
Made with Slashpage