Sign In

Distribution-Free Uncertainty Quantification for Continuous AI Agent Evaluation

์ž‘์„ฑ์ž
  • Haebom
์นดํ…Œ๊ณ ๋ฆฌ
Empty

์ €์ž

Yuxuan Gao, Megan Wang, Yi Ling Yu

๐Ÿ’ก ๊ฐœ์š”

์ด ๋…ผ๋ฌธ์€ AI ์—์ด์ „ํŠธ์˜ ์ง€์†์ ์ธ ํ‰๊ฐ€๋ฅผ ์œ„ํ•œ ๋ถ„ํฌ-์ž์œ  ๋ถˆํ™•์‹ค์„ฑ ์ •๋Ÿ‰ํ™” ๋ฐฉ๋ฒ•์„ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. ๋ถ„ํ•  ๋™ํ˜• ์˜ˆ์ธก(split conformal prediction) ๋ฐ ์ ์‘ํ˜• ๋™ํ˜• ์ถ”๋ก (ACI)์„ ์‚ฌ์šฉํ•˜์—ฌ ์˜ˆ์ธก ํ’ˆ์งˆ ์ ์ˆ˜์— ๋Œ€ํ•œ ๋ถ„ํฌ-์ž์œ  ์ปค๋ฒ„๋ฆฌ์ง€ ๋ณด์žฅ์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค. ์ œ์•ˆ๋œ ๋ฐฉ๋ฒ•์€ 24์‹œ๊ฐ„ ์˜ˆ์ธก ์‹œ 0.02 ๋ฏธ๋งŒ์˜ ๋ณด์ • ์˜ค๋ฅ˜๋ฅผ ๋‹ฌ์„ฑํ•˜๋ฉฐ, ์—์ด์ „ํŠธ ์ถœ์‹œ ํ›„ ๊ฐ„๊ฒฉ์„ 35% ๋„“ํžˆ๊ณ  ์žฌ์ˆ˜๋ ดํ•˜๋Š” ACI์˜ ํšจ๊ณผ๋ฅผ ์ž…์ฆํ•ฉ๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
AI ์—์ด์ „ํŠธ์˜ ์„ฑ๋Šฅ ์˜ˆ์ธก์— ๋Œ€ํ•œ ์‹ ๋ขฐํ•  ์ˆ˜ ์žˆ๋Š” ๋ถˆํ™•์‹ค์„ฑ ์ธก์ • ๊ธฐ์ค€์„ ์ œ๊ณตํ•ฉ๋‹ˆ๋‹ค.
โ€ข
๋‹ค์ค‘ ์—์ด์ „ํŠธ ํŒŒ์ดํ”„๋ผ์ธ ๋ฐ ์ˆœ์œ„ ๊ธฐ๋ฐ˜ ํ‰๊ฐ€๋ฅผ ์œ„ํ•œ ์ƒˆ๋กœ์šด ๋ถˆํ™•์‹ค์„ฑ ๊ฒฝ๊ณ„ ๋ฐ ๊ทœ์น™์„ ๊ฐœ๋ฐœํ–ˆ์Šต๋‹ˆ๋‹ค.
โ€ข
์‹ค์ œ ์‹œ๋ฎฌ๋ ˆ์ด์…˜ ๋ฐ ๋ฐ์ดํ„ฐ ํ‰๊ฐ€๋ฅผ ํ†ตํ•ด ์ œ์•ˆ๋œ ๋ฐฉ๋ฒ•์˜ ํšจ๊ณผ๋ฅผ ์ž…์ฆํ•ฉ๋‹ˆ๋‹ค.
โ€ข
์ œ์•ˆ๋œ ํ”„๋ ˆ์ž„์›Œํฌ์˜ ์ผ๋ฐ˜ํ™” ๊ฐ€๋Šฅ์„ฑ ๋ฐ ๋Œ€๊ทœ๋ชจ ์‹œ์Šคํ…œ ์ ์šฉ ์‹œ์˜ ํšจ์œจ์„ฑ์— ๋Œ€ํ•œ ์ถ”๊ฐ€์ ์ธ ์—ฐ๊ตฌ๊ฐ€ ํ•„์š”ํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
๐Ÿ‘
Made with Slashpage