Daily Arxiv

์ „ ์„ธ๊ณ„์—์„œ ๋ฐœ๊ฐ„๋˜๋Š” ์ธ๊ณต์ง€๋Šฅ ๊ด€๋ จ ๋…ผ๋ฌธ์„ ์ •๋ฆฌํ•˜๋Š” ํŽ˜์ด์ง€ ์ž…๋‹ˆ๋‹ค.
๋ณธ ํŽ˜์ด์ง€๋Š” Google Gemini๋ฅผ ํ™œ์šฉํ•ด ์š”์•ฝ ์ •๋ฆฌํ•˜๋ฉฐ, ๋น„์˜๋ฆฌ๋กœ ์šด์˜ ๋ฉ๋‹ˆ๋‹ค.
๋…ผ๋ฌธ์— ๋Œ€ํ•œ ์ €์ž‘๊ถŒ์€ ์ €์ž ๋ฐ ํ•ด๋‹น ๊ธฐ๊ด€์— ์žˆ์œผ๋ฉฐ, ๊ณต์œ  ์‹œ ์ถœ์ฒ˜๋งŒ ๋ช…๊ธฐํ•˜๋ฉด ๋ฉ๋‹ˆ๋‹ค.

Zoom in, Click out: Unlocking and Evaluating the Potential of Zooming for GUI Grounding

Created by
  • Haebom
Category
Empty

์ €์ž

Zhiyuan Jiang, Shenghao Xie, Wenyi Li, Wenqiang Zu, Peihang Li, Jiahao Qiu, Siqi Pei, Lei Ma, Tiejun Huang, Mengdi Wang, Shilong Liu

๐Ÿ’ก ๊ฐœ์š”

๋ณธ ๋…ผ๋ฌธ์€ GUI ์—์ด์ „ํŠธ ๊ตฌ์ถ•์˜ ํ•ต์‹ฌ ๊ธฐ์ˆ ์ธ GUI grounding์„ ์œ„ํ•œ ์ƒˆ๋กœ์šด ๋ฐฉ๋ฒ•์œผ๋กœ ์คŒ ๊ธฐ๋Šฅ์„ ํ™œ์šฉํ•˜๋Š” 'ZoomClick'์„ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. ZoomClick์€ ์คŒ์˜ 4๊ฐ€์ง€ ์ฃผ์š” ์†์„ฑ์„ ํ™œ์šฉํ•˜์—ฌ, ๋ณ„๋„์˜ ํ•™์Šต ์—†์ด GUI grounding ๋ชจ๋ธ์˜ ์„ฑ๋Šฅ์„ ํ–ฅ์ƒ์‹œํ‚ต๋‹ˆ๋‹ค. ์‹คํ—˜ ๊ฒฐ๊ณผ, ZoomClick์€ ๊ธฐ์กด GUI grounding ๋ชจ๋ธ์˜ ์„ฑ๋Šฅ์„ ํฌ๊ฒŒ ํ–ฅ์ƒ์‹œ์ผฐ์œผ๋ฉฐ, ์—ฌ๋Ÿฌ ๋ฒค์น˜๋งˆํฌ์—์„œ ์ตœ๊ณ  ์ˆ˜์ค€์˜ ๊ฒฐ๊ณผ๋ฅผ ๋‹ฌ์„ฑํ–ˆ์Šต๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
์คŒ ๊ธฐ๋Šฅ์„ ํ™œ์šฉํ•˜์—ฌ ๋ณ„๋„์˜ ํ•™์Šต ์—†์ด GUI grounding ๋ชจ๋ธ์˜ ์„ฑ๋Šฅ์„ ํšจ๊ณผ์ ์œผ๋กœ ํ–ฅ์ƒ์‹œํ‚ฌ ์ˆ˜ ์žˆ์Œ์„ ์ž…์ฆํ–ˆ์Šต๋‹ˆ๋‹ค.
โ€ข
์คŒ ๊ธฐ๋Šฅ์˜ ์ค‘์š”์„ฑ์„ ๊ฐ•์กฐํ•˜๊ณ , GUI grounding ์—ฐ๊ตฌ ๋ถ„์•ผ์˜ ์ƒˆ๋กœ์šด ๊ฐ€๋Šฅ์„ฑ์„ ์ œ์‹œํ–ˆ์Šต๋‹ˆ๋‹ค.
โ€ข
GUIZoom-Bench๋ผ๋Š” ์ƒˆ๋กœ์šด ๋ฒค์น˜๋งˆํฌ๋ฅผ ์ œ์‹œํ•˜์—ฌ ํ–ฅํ›„ ์คŒ ๊ธฐ๋ฐ˜ GUI grounding ์—ฐ๊ตฌ์˜ ๋ฐœ์ „์„ ์œ„ํ•œ ๊ธฐ๋ฐ˜์„ ๋งˆ๋ จํ–ˆ์Šต๋‹ˆ๋‹ค.
โ€ข
(ํ•œ๊ณ„์  ๋˜๋Š” ํ–ฅํ›„ ๊ณผ์ œ) ZoomClick์€ ์คŒ ๊ธฐ๋Šฅ์„ ๊ธฐ๋ฐ˜์œผ๋กœ ํ•˜๋ฉฐ, ์คŒ ๊ธฐ๋Šฅ์„ ์ง€์›ํ•˜์ง€ ์•Š๋Š” ํ™˜๊ฒฝ์—์„œ๋Š” ์ ์šฉ์ด ์–ด๋ ต์Šต๋‹ˆ๋‹ค. ํ–ฅํ›„์—๋Š” ์คŒ ๊ธฐ๋Šฅ ์—†์ด๋„ ์œ ์‚ฌํ•œ ์„ฑ๋Šฅ์„ ๋‚ผ ์ˆ˜ ์žˆ๋Š” ๋ฐฉ๋ฒ•๋ก  ๊ฐœ๋ฐœ ๋ฐ ๋‹ค์–‘ํ•œ ํ™˜๊ฒฝ์—์„œ์˜ ์ ์šฉ ๊ฐ€๋Šฅ์„ฑ์„ ๋†’์ด๋Š” ์—ฐ๊ตฌ๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
๐Ÿ‘