Sign In

Distributional Process Reward Models: Calibrated Prediction of Future Rewards via Conditional Optimal Transport

์ž‘์„ฑ์ž
  • Haebom
์นดํ…Œ๊ณ ๋ฆฌ
Empty

์ €์ž

Rachel Ma, Dylan Hadfield-Menell, Kristjan Greenewald

๐Ÿ’ก ๊ฐœ์š”

๊ธฐ์กด Process Reward Model (PRM)์€ ๋ฏธ๋ž˜ ์„ฑ๊ณต ํ™•๋ฅ ์„ ๊ณผ๋Œ€ํ‰๊ฐ€ํ•˜๋Š” ๊ฒฝํ–ฅ์ด ์žˆ์–ด ๋ณด์ •(calibration)์ด ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค. ๋ณธ ๋…ผ๋ฌธ์€ ์กฐ๊ฑด๋ถ€ ์ตœ์  ์šด์†ก(conditional optimal transport)์„ PRM ๋ณด์ •์— ์ตœ์ดˆ๋กœ ์ ์šฉํ•˜์—ฌ, PRM์˜ ์˜ˆ์ธก๊ฐ’์— ๋Œ€ํ•œ ์กฐ๊ฑด๋ถ€ ๋ถ„์œ„์ˆ˜ ํ•จ์ˆ˜๋ฅผ ์ถ”์ •ํ•ฉ๋‹ˆ๋‹ค. ์ด๋ฅผ ํ†ตํ•ด ์‹ ๋ขฐ ๊ตฌ๊ฐ„์„ ์œ ์—ฐํ•˜๊ฒŒ ์ถ”์ถœํ•˜๊ณ , ์ธ์Šคํ„ด์Šค ์ ์‘ํ˜• ์Šค์ผ€์ผ๋ง(IAS) ํ”„๋ ˆ์ž„์›Œํฌ์— ํ†ตํ•ฉํ•˜์—ฌ ์˜ˆ์ธก ์„ฑ๋Šฅ์„ ํ–ฅ์ƒ์‹œํ‚ต๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
PRM์˜ ์˜ˆ์ธก๊ฐ’์— ๋Œ€ํ•œ ๊ตฌ์กฐ์ ์œผ๋กœ ํƒ€๋‹นํ•œ ๋ถ„์œ„์ˆ˜ ์ถ”์ •์น˜๋ฅผ ์ œ๊ณตํ•˜์—ฌ ๋ณด์ • ์„ฑ๋Šฅ์„ ๊ฐœ์„ ํ•ฉ๋‹ˆ๋‹ค.
โ€ข
๋‹ค์–‘ํ•œ ์‹ ๋ขฐ ์ˆ˜์ค€์—์„œ ํšจ์œจ์ ์ธ ๋ถˆํ™•์‹ค์„ฑ ์ถ”์ •์น˜๋ฅผ ์ถ”์ถœํ•  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
โ€ข
์ˆ˜ํ•™์  ์ถ”๋ก  ๋ฒค์น˜๋งˆํฌ์—์„œ ๊ธฐ์กด ๋ณด์ • ๋ฐฉ๋ฒ• ๋Œ€๋น„ ์šฐ์ˆ˜ํ•œ ์„ฑ๋Šฅ์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.
โ€ข
PRM์˜ ์ˆœ์œ„ ์‹ ํ˜ธ๊ฐ€ ์‹ ๋ขฐํ•  ์ˆ˜ ์žˆ๋Š” ๊ฒฝ์šฐ์— ์„ฑ๋Šฅ์ด ํ–ฅ์ƒ๋˜๋Š” ๊ฒฝํ–ฅ์ด ์žˆ์–ด, ์‹ ํ˜ธ์˜ ์‹ ๋ขฐ์„ฑ์ด ์ค‘์š”ํ•œ ์ œ์•ฝ ์กฐ๊ฑด์ด ๋  ์ˆ˜ ์žˆ์Šต๋‹ˆ๋‹ค.
๐Ÿ‘
Made with Slashpage