We highlight the shortcomings of the traditional closed peer review system and study the efficacy and accuracy of an alternative peer review system based on an open, top-down process. Using data from two major scientific conferences (CCN2023 and ICLR2023), we highlight the high variability and low correlation in reviewer scores. Using a reviewer quality estimator, we reveal the lack of correlation between reviewer quality and author quality. Furthermore, we find that authors with moderate article scores are the best reviewers. Using a Bayesian method, we estimate article quality and demonstrate that reviewer evaluation in an open system can yield high-quality article scores. Finally, we propose an incentive structure to recognize high-quality reviewers and encourage broader review coverage.
