Sign In

Evaluating Deep Research Agents on Expert Consulting Work: A Benchmark with Verifiers, Rubrics, and Cognitive Traps

์ž‘์„ฑ์ž
  • Haebom
์นดํ…Œ๊ณ ๋ฆฌ
Empty

์ €์ž

Tanmay Asthana, Aman Saksena, Divyansh Sahu

๐Ÿ’ก ๊ฐœ์š”

์ด ๋…ผ๋ฌธ์€ ์‹ฌ์ธต ์—ฐ๊ตฌ ์—์ด์ „ํŠธ(DRA)์˜ ์‹ค์ œ ์—…๋ฌด ์ˆ˜ํ–‰ ๋Šฅ๋ ฅ์„ ํ‰๊ฐ€ํ•˜๊ธฐ ์œ„ํ•œ ์ƒˆ๋กœ์šด ๋ฒค์น˜๋งˆํฌ๋ฅผ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. ๊ธฐ์กด ๋ฒค์น˜๋งˆํฌ๊ฐ€ ๋‹จ์ˆœํ•œ ์‚ฌ์‹ค ๊ฒ€์ƒ‰ ๋Šฅ๋ ฅ๋งŒ ์ธก์ •ํ•˜๋Š” ๋ฐ˜๋ฉด, ์ด ๋ฒค์น˜๋งˆํฌ๋Š” ๋ณต์žกํ•œ ๋ฌธ์„œ ์ข…ํ•ฉ ๋ฐ ๊ตฌ์กฐํ™”๋œ ๊ฒฐ๊ณผ๋ฌผ ์ƒ์„ฑ์„ ์š”๊ตฌํ•˜๋Š” ๊ด€๋ฆฌ ์ปจ์„คํ„ดํŠธ ์—…๋ฌด๋ฅผ ๋ชจ๋ฐฉํ•ฉ๋‹ˆ๋‹ค. ์„ธ ๊ฐ€์ง€ ์„ ๋„์ ์ธ DRA(Claude Opus 4.6, OpenAI o3-deep-research, Google Gemini 3.1 Pro deep-research)๋ฅผ ํ‰๊ฐ€ํ•œ ๊ฒฐ๊ณผ, ์—„๊ฒฉํ•œ ๊ธฐ์ค€ ํ•˜์—์„œ ์ด๋“ค ์—์ด์ „ํŠธ์˜ ์ˆ˜์šฉ๋ฅ ์€ ์ „๋ฐ˜์ ์œผ๋กœ ๋งค์šฐ ๋‚ฎ์•˜์Šต๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
DRA์˜ ํ˜„์‹ค ์—…๋ฌด ์ˆ˜ํ–‰ ๋Šฅ๋ ฅ์— ๋Œ€ํ•œ ์—„๊ฒฉํ•œ ํ‰๊ฐ€ ๊ธฐ์ค€ ์ œ์‹œ: ๋ณต์žกํ•œ ๋ถ„์„ ๋ฐ ๊ตฌ์กฐํ™”๋œ ๊ฒฐ๊ณผ๋ฌผ ์ƒ์„ฑ์„ ์š”๊ตฌํ•˜๋Š” ์‹ค์ œ ์ปจ์„คํŒ… ์—…๋ฌด ํ™˜๊ฒฝ์—์„œ์˜ DRA ์„ฑ๋Šฅ์„ ์ธก์ •ํ•  ์ˆ˜ ์žˆ๋Š” ์ƒˆ๋กœ์šด ๋ฒค์น˜๋งˆํฌ๋ฅผ ๊ฐœ๋ฐœํ•˜์—ฌ, ๊ธฐ์กด ํ‰๊ฐ€ ๋ฐฉ์‹์˜ ํ•œ๊ณ„๋ฅผ ๊ทน๋ณตํ–ˆ์Šต๋‹ˆ๋‹ค.
โ€ข
DRA๋ณ„ ๊ฐ•์  ๋ฐ ์•ฝ์  ๋ถ„์„: Claude๋Š” ํŒŒ์ผ ๊ด€๋ จ ์ž‘์—…์—์„œ ์‹ ๋ขฐ๋„๊ฐ€ ๋†’์œผ๋‚˜ ํ™˜๊ฐ ํ˜„์ƒ์ด ๋‘๋“œ๋Ÿฌ์กŒ๊ณ , o3๋Š” ์ถ”๋ก  ๋Šฅ๋ ฅ์ด ์šฐ์ˆ˜ํ–ˆ์œผ๋‚˜ ๋ˆ„๋ฝ๋˜๋Š” ์„น์…˜๊ณผ ์‚ฐ์ˆ  ์˜ค๋ฅ˜๊ฐ€ ๋ฐœ์ƒํ–ˆ์Šต๋‹ˆ๋‹ค. Gemini๋Š” ๋†’์€ ์ˆ˜์šฉ๋ฅ ๊ณผ ํ•จ๊ป˜ ๊ฐ€์žฅ ๋งŽ์€ 0์  ์…€์„ ๊ธฐ๋กํ•˜๋Š” ์–‘๊ทนํ™”๋œ ์„ฑ๋Šฅ์„ ๋ณด์˜€์Šต๋‹ˆ๋‹ค.
โ€ข
DRA์˜ ์‹ค์ œ ์ ์šฉ์„ ์œ„ํ•œ ์ถ”๊ฐ€์ ์ธ ๋ฐœ์ „ ํ•„์š”: ํ˜„์žฌ DRA๋Š” ๋ณต์žกํ•˜๊ณ  ์˜์‚ฌ ๊ฒฐ์ •์ด ํ•„์š”ํ•œ ์—…๋ฌด๋ฅผ ์ˆ˜ํ–‰ํ•˜๊ธฐ์—๋Š” ์•„์ง ๋ถ€์กฑํ•˜๋ฉฐ, ํ™˜๊ฐ ํ˜„์ƒ, ๋…ผ๋ฆฌ ์˜ค๋ฅ˜, ๋ˆ„๋ฝ ๋“ฑ ํ•ด๊ฒฐํ•ด์•ผ ํ•  ๊ณผ์ œ๊ฐ€ ๋งŽ์Šต๋‹ˆ๋‹ค.
๐Ÿ‘
Made with Slashpage