Computer Science 5th Year Masters Thesis Presentation
— 3:30pm
Location:
In Person
-
Traffic21 Classroom, Gates Hillman 6501
Speaker:
JHIH-YI HSIEH
,
Masters Student, Computer Science Department, Carnegie Mellon University
https://www.linkedin.com/in/jhih-yi-hsieh-5931971a1
Automated Peer-Reviewer Assignment can be Manipulated to Secure Reviews from Colluders
Peer review has been the method of choice for academics to ensure the overall quality of published research. Conferences in computer science, which are typically the primary venues to publish research, have grown tremendously with over 10,000 papers submitted to some conferences. Consequently, many parts of the review process are now automated, notably the assignment of expert reviewers to the submitted papers. These automated assignments depend on, among other things, similarities computed between reviewers and papers using natural language processing (NLP) techniques. A higher similarity between a reviewer and a submitted paper means that this reviewer is envisaged to have higher expertise for the paper.
A major problem in peer review is the existence of collusion rings, where groups of researchers try to get assigned each others' papers and then provide positive reviews to each other irrespective of the true qualities. It is commonly believed that the NLP-matching component of reviewer assignments is safe from manipulation by collusion rings. Consequently, most work on defending or investigating collusion rings focuses on other components of the review process, assuming the NLP-matching is not manipulated.
In this thesis, we show that the automated reviewer assignments used by OpenReview, the main peer review platform for conferences in machine learning and related fields, can be manipulated to ensure fellow colluders get assigned to other colluders' papers. We demonstrate this susceptibility via methods to modify colluders' abstracts, by both manual and algorithmic means, that can fool the automated assignments. Our findings have significant implications for defending against fraudulent behavior in peer review, for fairness and integrity of science.
Thesis Committee:
Nihar Shah (Chair)
Aditi Raghunathan