Artificial Intelligence Seminar
— 1:00pm
Location:
In Person and Virtual - ET
-
ASA Conference Room, Gates Hillman 6115 and Zoom
Speaker:
RAFAEL M. FRONGILLO
,
Assistant Professor of Computing, Theoretical Computer Science, College of Engineering and Applied Science, University of Colorado, Boulder
https://home.cs.colorado.edu/~raf/
Incentive problems in data science competitions, and how to fix them
Machine learning and data science competitions, wherein contestants submit predictions about held-out data points, are an increasingly common way to gather information and identify experts. One of the most prominent platforms is Kaggle, which has run competitions with prizes up to 3 million USD. The traditional mechanism for selecting the winner is simple: score each prediction on each held-out data point, and the contestant with the highest total score wins. Perhaps surprisingly, this reasonable and popular mechanism can incentivize contestants to submit wildly inaccurate predictions.
The talk will begin with a series of experiments inspired by Aldous (2019) to build intuition for the incentive issues and what sort of strategic behavior one would expect---and when. One takeaway is that, despite conventional wisdom, large held-out data sets do not always alleviate these incentive issues, and small ones do not necessarily suffer from them, as we confirm with formal results. We will then discuss a new mechanism which is approximately truthful, in the sense that rational contestants will submit predictions which are close to their best guess. If time we will see how the same mechanism solves an open question for online learning from strategic experts.
—
Rafael (Raf) Frongillo is an Assistant Professor of Computer Science at the University of Colorado Boulder. His research lies at the interface between theoretical machine learning and economics, primarily focusing on information elicitation mechanisms, which incentivize humans or algorithms to predict accurately. Before Boulder, Raf was a postdoc at the Center for Research on Computation and Society at Harvard University and at Microsoft Research New York. He received his PhD in Computer Science at UC Berkeley, advised by Christos Papadimitriou and supported by the NDSEG Fellowship.
The AI Seminar is sponsored by SambaNova Systems.
In Person and Zoom Participation. See announcement.
Event Website:
http://www.cs.cmu.edu/~aiseminar/