Sign In

ReplicatorBench: Benchmarking LLM Agents for Replicability in Social and Behavioral Sciences

Created by
  • Haebom
Category
Empty

์ €์ž

Bang Nguyen, Dominik Soos, Qian Ma, Rochana R. Obadage, Zack Ranjan, Sai Koneru, Anna Szabelska, Adam Gill, Timothy M. Errington, Shakhlo Nematova, Sarah Rajtmajer, Jian Wu, Meng Jiang

๐Ÿ’ก ๊ฐœ์š”

๋ณธ ๋…ผ๋ฌธ์€ ์‚ฌํšŒ๊ณผํ•™ ๋ฐ ํ–‰๋™๊ณผํ•™ ๋ถ„์•ผ์—์„œ ์ธ๊ฐ„ ์—ฐ๊ตฌ์ž์˜ ๋ณต์ œ ๊ณผ์ •(replication process)์„ ๋ชจ๋ฐฉํ•  ์ˆ˜ ์žˆ๋Š” AI ์—์ด์ „ํŠธ๋ฅผ ํ‰๊ฐ€ํ•˜๊ธฐ ์œ„ํ•œ ์ƒˆ๋กœ์šด ๋ฒค์น˜๋งˆํฌ์ธ ReplicatorBench๋ฅผ ์ œ์•ˆํ•ฉ๋‹ˆ๋‹ค. ReplicatorBench๋Š” ๋ณต์ œ ๊ฐ€๋Šฅํ•œ ์—ฐ๊ตฌ์™€ ๋ณต์ œ ๋ถˆ๊ฐ€๋Šฅํ•œ ์—ฐ๊ตฌ ๋ชจ๋‘๋ฅผ ํฌํ•จํ•˜๋ฉฐ, ๋ฐ์ดํ„ฐ ์ถ”์ถœ, ์‹คํ—˜ ์„ค๊ณ„ ๋ฐ ์‹คํ–‰, ๊ฒฐ๊ณผ ํ•ด์„์˜ ์„ธ ๋‹จ๊ณ„๋ฅผ ํ‰๊ฐ€ํ•ฉ๋‹ˆ๋‹ค. ํ˜„์žฌ LLM ๊ธฐ๋ฐ˜ ์—์ด์ „ํŠธ๋Š” ์‹คํ—˜ ์„ค๊ณ„ ๋ฐ ์‹คํ–‰์— ๊ฐ•์ ์„ ๋ณด์ด์ง€๋งŒ, ๋ณต์ œ์— ํ•„์š”ํ•œ ์ƒˆ๋กœ์šด ๋ฐ์ดํ„ฐ์™€ ๊ฐ™์€ ์ž๋ฃŒ๋ฅผ ๊ฒ€์ƒ‰ํ•˜๋Š” ๋ฐ ์–ด๋ ค์›€์„ ๊ฒช๋Š” ๊ฒƒ์œผ๋กœ ๋‚˜ํƒ€๋‚ฌ์Šต๋‹ˆ๋‹ค.

๐Ÿ”‘ ์‹œ์‚ฌ์  ๋ฐ ํ•œ๊ณ„

โ€ข
AI ์—์ด์ „ํŠธ๊ฐ€ ์‹ค์ œ ์—ฐ๊ตฌ ๋ณต์ œ์™€ ์œ ์‚ฌํ•œ ๋‹ค๋‹จ๊ณ„ ํ”„๋กœ์„ธ์Šค๋ฅผ ์ˆ˜ํ–‰ํ•  ์ˆ˜ ์žˆ๋Š” ์ž ์žฌ๋ ฅ์„ ๋ณด์—ฌ์ค๋‹ˆ๋‹ค.
โ€ข
๋ณต์ œ ๋ถˆ๊ฐ€๋Šฅํ•œ ์—ฐ๊ตฌ๋ฅผ ์‹๋ณ„ํ•˜๋Š” AI ์—์ด์ „ํŠธ์˜ ๋Šฅ๋ ฅ ํ‰๊ฐ€์— ๋Œ€ํ•œ ์ค‘์š”์„ฑ์„ ๊ฐ•์กฐํ•ฉ๋‹ˆ๋‹ค.
โ€ข
ํ˜„์žฌ LLM ์—์ด์ „ํŠธ์˜ ๋ฐ์ดํ„ฐ ๊ฒ€์ƒ‰ ๋Šฅ๋ ฅ ๋ถ€์กฑ์€ ๊ฐœ์„ ์ด ํ•„์š”ํ•œ ์ฃผ์š” ์˜์—ญ์ž„์„ ์‹œ์‚ฌํ•ฉ๋‹ˆ๋‹ค.
โ€ข
ReplicatorBench์˜ ํ‰๊ฐ€ ๋Œ€์ƒ์ด ์ฃผ๋กœ ์‚ฌํšŒ๊ณผํ•™ ๋ฐ ํ–‰๋™๊ณผํ•™ ๋ถ„์•ผ์— ๊ตญํ•œ๋˜์–ด ์žˆ์–ด, ๋‹ค๋ฅธ ๊ณผํ•™ ๋ถ„์•ผ๋กœ์˜ ์ผ๋ฐ˜ํ™” ๊ฐ€๋Šฅ์„ฑ์— ๋Œ€ํ•œ ์ถ”๊ฐ€ ์—ฐ๊ตฌ๊ฐ€ ํ•„์š”ํ•ฉ๋‹ˆ๋‹ค.
๐Ÿ‘
Made with Slashpage