Sign In

Mixture of Layers with Hybrid Attention

์ž‘์„ฑ์ž
  • Haebom
์นดํ…Œ๊ณ ๋ฆฌ
Empty

์ €์ž

Ivan Ternovtsii, Yurii Bilak

๐Ÿ’ก ๊ฐœ์š”

๋ณธ ๋…ผ๋ฌธ์€ ๊ธฐ์กด MoE ๋ชจ๋ธ์—์„œ ๊ฐ ๋ ˆ์ด์–ด๋Š” ๋‹จ์ผ ๋ธ”๋ก์œผ๋กœ ๊ตฌ์„ฑ๋˜๋Š” ๋ฌธ์ œ์ ์„ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด, ์—ฌ๋Ÿฌ ๊ฐœ์˜ ์–‡์€ ๋ธ”๋ก๋“ค์„ ๋ณ‘๋ ฌ๋กœ ๊ตฌ์„ฑํ•˜๊ณ  ๋ผ์šฐํŒ… ๋ฉ”์ปค๋‹ˆ์ฆ˜์„ ํ†ตํ•ด ํ† ํฐ์„ ํ• ๋‹นํ•˜๋Š” Mixture of Layers (MoL)๋ฅผ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. MoL์€ sparse attention์œผ๋กœ ์ธํ•œ ํ† ํฐ ์ปค๋ฒ„๋ฆฌ์ง€ ๋ถ€์กฑ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด, ๊ณต์œ ๋œ ๊ธ€๋กœ๋ฒŒ ์ปจํ…์ŠคํŠธ์šฉ softmax ๋ธ”๋ก๊ณผ ๊ฐ ๋ผ์šฐํŒ…๋œ ๋ธ”๋ก์— gated delta linear attention์„ ๊ฒฐํ•ฉํ•œ hybrid attention์„ ๋„์ž…ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ๋” ํšจ์œจ์ ์ธ ๊ณ„์‚ฐ๊ณผ ํ–ฅ์ƒ๋œ ์„ฑ๋Šฅ์„ ๋‹ฌ์„ฑํ•ฉ๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
MoE ๋ชจ๋ธ์˜ ๋ธ”๋ก ๋‹จ์œ„ ํฌ์†Œ์„ฑ์„ ๋ ˆ์ด์–ด ๋‹จ์œ„๋กœ ํ™•์žฅํ•˜์—ฌ ๋ชจ๋ธ์˜ ํ‘œํ˜„๋ ฅ์„ ๋†’์ผ ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
โ€ข
Hybrid attention์€ ๊ธ€๋กœ๋ฒŒ ์ปจํ…์ŠคํŠธ์™€ ์ง€์—ญ์  ํŠน์„ฑ์„ ๋™์‹œ์— ํฌ์ฐฉํ•˜์—ฌ sparse attention์˜ ๋‹จ์ ์„ ๋ณด์™„ํ•ฉ๋‹ˆ๋‹ค.
โ€ข
๋งŽ์€ ์ˆ˜์˜ ๋ธ”๋ก์„ ๋ผ์šฐํŒ…ํ•  ๋•Œ ๋ฐœ์ƒํ•˜๋Š” attention coverage ๋ฌธ์ œ๋ฅผ ํšจ๊ณผ์ ์œผ๋กœ ํ•ด๊ฒฐํ–ˆ์Šต๋‹ˆ๋‹ค.
โ€ข
Hybrid attention ๋ฉ”์ปค๋‹ˆ์ฆ˜์˜ ๋ณต์žก์„ฑ์ด ๋ชจ๋ธ ๊ตฌํ˜„ ๋ฐ ํŠœ๋‹์— ์ถ”๊ฐ€์ ์ธ ๊ณ ๋ ค์‚ฌํ•ญ์„ ์š”๊ตฌํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
โ€ข
์ œ์•ˆ๋œ MoL ๊ตฌ์กฐ๊ฐ€ ๋‹ค์–‘ํ•œ ํƒœ์Šคํฌ ๋ฐ ๋ฐ์ดํ„ฐ์…‹์—์„œ ์–ผ๋งˆ๋‚˜ ์ผ๋ฐ˜ํ™”๋  ์ˆ˜ ์žˆ๋Š”์ง€์— ๋Œ€ํ•œ ์ถ”๊ฐ€์ ์ธ ๊ฒ€์ฆ์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
๐Ÿ‘
Made with Slashpage