Sign In

Optimizing LLM Inference: Fluid-Guided Online Scheduling with Memory Constraints

์ž‘์„ฑ์ž
  • Haebom
์นดํ…Œ๊ณ ๋ฆฌ
Empty

์ €์ž

Ruicheng Ao, Gan Luo, David Simchi-Levi, Xinshang Wang

๐Ÿ’ก ๊ฐœ์š”

๋Œ€๊ทœ๋ชจ ์–ธ์–ด ๋ชจ๋ธ(LLM)์˜ ์ถ”๋ก  ๋น„์šฉ์ด ๋งค์šฐ ๋†’์œผ๋ฉฐ, GPU ๋ฉ”๋ชจ๋ฆฌ ์ œ์•ฝ์œผ๋กœ ์ธํ•œ KV ์บ์‹œ ์ฆ๊ฐ€ ๋ฌธ์ œ๋Š” ์ฒ˜๋ฆฌ๋Ÿ‰๊ณผ ๋น„์šฉ์— ํฐ ์˜ํ–ฅ์„ ๋ฏธ์นฉ๋‹ˆ๋‹ค. ๋ณธ ๋…ผ๋ฌธ์€ ๋ฉ”๋ชจ๋ฆฌ ์ œ์•ฝ ํ•˜์—์„œ LLM ์ถ”๋ก ์„ ์œ„ํ•œ ์˜จ๋ผ์ธ ์Šค์ผ€์ค„๋ง ๋ฌธ์ œ๋ฅผ ๋‹ค๋ฃจ๋ฉฐ, ์œ ์ฒด ๋ชจ๋ธ์„ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•œ WAIT ๋ฐ Nested WAIT ์•Œ๊ณ ๋ฆฌ์ฆ˜์„ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. ์ด ์•Œ๊ณ ๋ฆฌ์ฆ˜๋“ค์€ ๋ฉ”๋ชจ๋ฆฌ ์ดˆ๊ณผ๋กœ ์ธํ•œ ์š”์ฒญ ์ข…๋ฃŒ๋ฅผ ์ค„์ด๊ณ , ํŠนํžˆ ๋ถ€ํ•˜๊ฐ€ ๋†’๊ฑฐ๋‚˜ ์ดˆ๊ณผ๋œ ์ƒํ™ฉ์—์„œ ์ง€์—ฐ ์‹œ๊ฐ„์„ ๊ฐ์†Œ์‹œ์ผœ ์•ˆ์ •์ ์ธ ์šด์˜ ๋ฒ”์œ„๋ฅผ ๋„“ํž™๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
LLM ์ถ”๋ก  ๊ณผ์ •์—์„œ์˜ ๋™์  ๋ฉ”๋ชจ๋ฆฌ ์ฆ๊ฐ€(KV ์บ์‹œ) ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•œ ์ด๋ก ์  ํ”„๋ ˆ์ž„์›Œํฌ์™€ ์‹ค์šฉ์ ์ธ ์•Œ๊ณ ๋ฆฌ์ฆ˜์„ ์ œ์‹œํ•ฉ๋‹ˆ๋‹ค.
โ€ข
์ œ์•ˆ๋œ WAIT ๋ฐ Nested WAIT ์•Œ๊ณ ๋ฆฌ์ฆ˜์€ ๊ธฐ์กด ๋ฐฉ์‹ ๋Œ€๋น„ LLM ์ถ”๋ก  ์„œ๋น„์Šค์˜ ์•ˆ์ •์ ์ธ ์šด์˜ ๋ฒ”์œ„๋ฅผ ํ™•์žฅํ•˜๊ณ  ์„ฑ๋Šฅ์„ ๊ฐœ์„ ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
โ€ข
์˜ˆ์ธก ๋ถˆ๊ฐ€๋Šฅํ•œ ์ถœ๋ ฅ ๊ธธ์ด๋ฅผ ๊ฐ€์ง„ ์š”์ฒญ์— ๋Œ€ํ•œ Nested WAIT์˜ ์„ฑ๋Šฅ์€ ์ถ”๊ฐ€์ ์ธ ์•ˆ์ „ ๋ฒ„ํผ์— ์˜์กดํ•˜๋ฉฐ, ํ•ด๋‹น ๋ฒ„ํผ ํฌ๊ธฐ ์ตœ์ ํ™” ๋ฐ ๋‹ค์–‘ํ•œ ๋ชจ๋ธ/ํ•˜๋“œ์›จ์–ด ํ™˜๊ฒฝ์—์„œ์˜ ์ถ”๊ฐ€์ ์ธ ๊ฒ€์ฆ์ด ํ•„์š”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
๐Ÿ‘
Made with Slashpage