Computer Science Thesis Oral
— 1:00pm
Location:
In Person and Virtual - ET
-
Reddy Conference Room, Gates Hillman 4405 and Zoom
Speaker:
STEVEN JECMEN
,
Ph.D. Candidate, Computer Science Department, Carnegie Mellon University
https://sjecmen.github.io/
Making Peer Review Robust to Undesirable Behavior
Scientific peer review is a critical part of the academic publication process, used across disciplines and venues in various forms. Peer review generally relies on the good-faith participation of many reviewers and authors. However, the peer review process must also deal with different kinds of undesirable behavior from participants, including both malicious attempts to cheat the system and non-malicious cases of unreliability.
In this thesis, I describe several practical methods that we have proposed for handling different forms of undesirable behavior in peer review. First, we consider the problem of reviewer-author collusion, in which malicious reviewers manipulate the paper assignment in order to get assigned to each others' papers so that they can give them positive reviews. We provide efficient algorithms for finding high-quality randomized assignments that limit the probability that a colluding reviewer-author pair succeeds at manipulating the paper assignment.
Second, we provide an in-depth analysis of the cost of deploying a randomized assignment in terms of the resulting review quality. We propose methods that leverage the randomness introduced by these randomized assignments in order to evaluate alternative paper assignment policies, and apply these methods to estimate the quality of various potential changes to the assignment policy.
Finally, we analyze other approaches to addressing the manipulation of paper assignments, which we categorize into detection-based and mitigation-based approaches. We empirically analyze the problem of explicitly detecting reviewer-author collusion rings from the manipulated paper bidding. We also compare the tradeoffs between various proposed approaches to mitigating the impact of manipulated bidding. In addition, this talk will briefly discuss other forms of undesirable behavior addressed by this thesis: the issue of unresponsive reviewers, and the problem of “strategic reviewing,” in which reviewers give low scores to their assigned papers in the hopes of increasing their own paper’s chances of acceptance.
Thesis Committee:
Nihar Shah (Co-chair)
Fei Fang (Co-chair)
Christos Faloutsos
Yiling Chen (Harvard University)
Ashish Goel (Stanford University)
In Person and Zoom Participation. See announcement.