Sign In

AllMem: A Memory-centric Recipe for Efficient Long-context Modeling

Created by
  • Haebom
Category
Empty

์ €์ž

Ziming Wang, Xiang Wang, Kailong Peng, Lang Qin, Juan Gabriel Kostelec, Christos Sourmpis, Axel Laborieux, Qinghai Guo

๐Ÿ’ก ๊ฐœ์š”

๋ณธ ๋…ผ๋ฌธ์€ ๊ธด ์‹œํ€€์Šค ๋ชจ๋ธ๋ง์—์„œ ๋ฐœ์ƒํ•˜๋Š” LLM์˜ ๊ณ„์‚ฐ ๋ณต์žก์„ฑ๊ณผ ๋ฉ”๋ชจ๋ฆฌ ์˜ค๋ฒ„ํ—ค๋“œ ๋ฌธ์ œ๋ฅผ ํ•ด๊ฒฐํ•˜๊ธฐ ์œ„ํ•ด ์Šฌ๋ผ์ด๋”ฉ ์œˆ๋„์šฐ ์–ดํ…์…˜(SWA)๊ณผ ๋น„์„ ํ˜• ํ…Œ์ŠคํŠธ ์‹œ์  ํ›ˆ๋ จ(TTT) ๋ฉ”๋ชจ๋ฆฌ ๋„คํŠธ์›Œํฌ๋ฅผ ๊ฒฐํ•ฉํ•œ ํšจ์œจ์ ์ธ ํ•˜์ด๋ธŒ๋ฆฌ๋“œ ์•„ํ‚คํ…์ฒ˜์ธ AllMem์„ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. AllMem์€ ๋ชจ๋ธ์ด ์ดˆ์žฅ๊ธฐ ๋ฌธ๋งฅ์„ ํšจ๊ณผ์ ์œผ๋กœ ์ฒ˜๋ฆฌํ•˜๊ณ  ์น˜๋ช…์ ์ธ ๋ง๊ฐ์„ ์™„ํ™”ํ•˜๋ฉฐ, ๊ณ„์‚ฐ ๋ฐ ๋ฉ”๋ชจ๋ฆฌ ๋ถ€๋‹ด์„ ์ค„์ž…๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
๊ธฐ์กด LLM์˜ ์žฅ๊ธฐ ๋ฌธ๋งฅ ์ฒ˜๋ฆฌ ๋Šฅ๋ ฅ ๋ถ€์กฑ ๋ฐ ๋†’์€ ๊ณ„์‚ฐ/๋ฉ”๋ชจ๋ฆฌ ์š”๊ตฌ๋Ÿ‰ ๋ฌธ์ œ๋ฅผ ํšจ๊ณผ์ ์œผ๋กœ ํ•ด๊ฒฐํ•ฉ๋‹ˆ๋‹ค.
โ€ข
AllMem ์•„ํ‚คํ…์ฒ˜๋Š” ๊ธฐ์กด ์‚ฌ์ „ ํ•™์Šต๋œ LLM์„ ํšจ์œจ์ ์œผ๋กœ ๋ณ€ํ™˜ํ•  ์ˆ˜ ์žˆ์–ด ์ ์šฉ์„ฑ์ด ๋†’์Šต๋‹ˆ๋‹ค.
โ€ข
4k ์œˆ๋„์šฐ ๋ชจ๋ธ์€ 37k LongBench์—์„œ ๊ฑฐ์˜ ์†์‹ค ์—†๋Š” ์„ฑ๋Šฅ์„ ๋ณด์˜€๊ณ , 8k ์œˆ๋„์šฐ ๋ชจ๋ธ์€ 128k ๋ฌธ๋งฅ์—์„œ ๊ธฐ์กด ์–ดํ…์…˜๋ณด๋‹ค ์šฐ์ˆ˜ํ•œ ์„ฑ๋Šฅ์„ ๋ณด์—ฌ ํŒŒ๋ผ๋ฏธํ„ฐํ™”๋œ ๋ฉ”๋ชจ๋ฆฌ์˜ ํšจ์šฉ์„ฑ์„ ์ž…์ฆํ–ˆ์Šต๋‹ˆ๋‹ค.
โ€ข
(ํ•œ๊ณ„์  ๋˜๋Š” ํ–ฅํ›„ ๊ณผ์ œ) AllMem ์•„ํ‚คํ…์ฒ˜์˜ ์ตœ์  ์œˆ๋„์šฐ ํฌ๊ธฐ ๋ฐ ๋ฉ”๋ชจ๋ฆฌ ์šฉ๋Ÿ‰ ๊ฒฐ์ •, ๋‹ค์–‘ํ•œ LLM ์•„ํ‚คํ…์ฒ˜์— ๋Œ€ํ•œ ์ผ๋ฐ˜ํ™” ์„ฑ๋Šฅ ๊ฒ€์ฆ ๋“ฑ์ด ํ–ฅํ›„ ์—ฐ๊ตฌ ๊ณผ์ œ๋กœ ๋‚จ์Šต๋‹ˆ๋‹ค.
๐Ÿ‘
Made with Slashpage