Sign In

Text2GQL-Bench: A Text to Graph Query Language Benchmark [Experiment, Analysis & Benchmark]

Created by
  • Haebom
Category
Empty

์ €์ž

Songlin Lyu, Lujie Ban, Zihang Wu, Tianqi Luo, Jirong Liu, Chenhao Ma, Yuyu Luo, Nan Tang, Shipeng Qi, Heng Lin, Yongchao Liu, Chuntao Hong

๐Ÿ’ก ๊ฐœ์š”

์ด ์—ฐ๊ตฌ๋Š” ์ž์—ฐ์–ด ์งˆ์˜๋ฅผ ๊ทธ๋ž˜ํ”„ ์งˆ์˜ ์–ธ์–ด(GQL)๋กœ ๋ณ€ํ™˜ํ•˜๋Š” Text-to-GQL ์‹œ์Šคํ…œ์˜ ๋ฐœ์ „์„ ์ €ํ•ดํ•˜๋Š” ๊ธฐ์กด ๋ฐ์ดํ„ฐ์…‹์˜ ํ•œ๊ณ„๋ฅผ ๊ทน๋ณตํ•˜๊ธฐ ์œ„ํ•ด Text2GQL-Bench๋ผ๋Š” ํ†ตํ•ฉ ๋ฒค์น˜๋งˆํฌ๋ฅผ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. 13๊ฐœ ๋„๋ฉ”์ธ์— ๊ฑธ์ณ 178,184๊ฐœ์˜ ์งˆ์˜-์‘๋‹ต ์Œ์œผ๋กœ ๊ตฌ์„ฑ๋œ ๋ฉ€ํ‹ฐ-GQL ๋ฐ์ดํ„ฐ์…‹๊ณผ ๋‹ค์–‘ํ•œ ์กฐ๊ฑด์—์„œ ๋ฐ์ดํ„ฐ์…‹์„ ์ƒ์„ฑํ•˜๋Š” ํ”„๋ ˆ์ž„์›Œํฌ๋ฅผ ๊ตฌ์ถ•ํ–ˆ์Šต๋‹ˆ๋‹ค. ๋˜ํ•œ, ๋ฌธ๋ฒ•์  ์œ ํšจ์„ฑ, ์œ ์‚ฌ๋„, ์˜๋ฏธ๋ก ์  ์ผ์น˜, ์‹คํ–‰ ์ •ํ™•๋„๋ฅผ ์ข…ํ•ฉ์ ์œผ๋กœ ํ‰๊ฐ€ํ•˜๋Š” ๋ฐฉ๋ฒ•์„ ๋„์ž…ํ•˜์—ฌ ๋ชจ๋ธ์˜ ์„ฑ๋Šฅ์„ ์ฒด๊ณ„์ ์œผ๋กœ ์ธก์ •ํ•ฉ๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
LLM์ด ISO-GQL์„ ์ƒ์„ฑํ•˜๋Š” ๋ฐ ์žˆ์–ด '๋ฐฉ์–ธ(dialect)' ๊ฐ„์˜ ํฐ ๊ฒฉ์ฐจ๊ฐ€ ์กด์žฌํ•˜๋ฉฐ, ํŠนํžˆ ์ œ๋กœ์ƒท(zero-shot) ํ™˜๊ฒฝ์—์„œ๋Š” ์‹คํ–‰ ์ •ํ™•๋„๊ฐ€ ๋งค์šฐ ๋‚ฎ์Œ์„ ํ™•์ธํ–ˆ์Šต๋‹ˆ๋‹ค.
โ€ข
3-์ƒท(3-shot) ํ”„๋กฌํ”„ํŠธ๋ฅผ ์‚ฌ์šฉํ•˜๊ฑฐ๋‚˜ ๋ชจ๋ธ์„ ์ถฉ๋ถ„ํ•œ GQL ์˜ˆ์ œ๋กœ ํŒŒ์ธํŠœ๋‹(fine-tuning)ํ•˜๋Š” ๊ฒƒ์ด ์„ฑ๋Šฅ ํ–ฅ์ƒ์— ๊ฒฐ์ •์ ์ธ ์—ญํ• ์„ ํ•œ๋‹ค๋Š” ๊ฒƒ์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.
โ€ข
Text2GQL-Bench๋Š” ๋‹ค์–‘ํ•œ GQL๊ณผ ๋„๋ฉ”์ธ์— ๊ฑธ์ณ Text-to-GQL ์‹œ์Šคํ…œ์˜ ์„ฑ๋Šฅ์„ ํฌ๊ด„์ ์œผ๋กœ ํ‰๊ฐ€ํ•˜๊ณ  ๋น„๊ตํ•  ์ˆ˜ ์žˆ๋Š” ํ‘œ์ค€ํ™”๋œ ๋ฐฉ๋ฒ•์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
โ€ข
๋ฒค์น˜๋งˆํฌ์˜ ๊ทœ๋ชจ์™€ ๋‹ค์–‘์„ฑ์„ ๋”์šฑ ํ™•์žฅํ•˜๊ณ , ๋ณด๋‹ค ๋ณต์žกํ•˜๊ณ  ์‹ค์ œ์ ์ธ ๊ทธ๋ž˜ํ”„ ์งˆ์˜ ํŒจํ„ด์„ ํฌํ•จํ•˜๋Š” ๋ฐฉํ–ฅ์œผ๋กœ ์—ฐ๊ตฌ๋ฅผ ๋ฐœ์ „์‹œํ‚ฌ ํ•„์š”๊ฐ€ ์žˆ์Šต๋‹ˆ๋‹ค.
๐Ÿ‘
Made with Slashpage