Jingyan Wang is a Research Assistant Professor at the Toyota Technological Institute at Chicago. She was previously a postdoctoral fellow at Georgia Tech, affiliated with the School of Industrial and Systems Engineering and the Algorithm and Randomness Center. She received her PhD in Computer Science from Carnegie Mellon University and her B.S. in Electrical Engineering and Computer Sciences from UC Berkeley. She uses tools from statistics and machine learning to understand and improve evaluation systems, such as those involving hiring, admissions, grading, and peer review. Her interdisciplinary research has been published in machine learning, artificial intelligence, human computation, and economics and computation. She was the recipient of the Best Student Paper Award at AAMAS 2019, and was selected as a Rising Star in EECS and in Data Science.
Talk: Mitigating Biases in Evaluation: Offline and Online Settings
Abstract: Evaluation -- to estimate the quality of items or people -- is central to many real-world applications such as admissions, grading, hiring, and peer review. In this talk, I will present two vignettes of my research towards understanding and mitigating various sources of biases in evaluation.
(1) Outcome-induced bias: We consider how people’s ratings are affected by experiences irrelevant to the evaluation objective. For example, in teaching evaluation, students who receive higher grades tend to rate their instructors more positively. In such scenarios, we propose mild non-parametric assumptions to model the bias, design an adaptive correction algorithm, and prove its consistency guarantees.
(2) Symbiosis bias: We consider A/B testing that aims to compare the performance of a pair of online algorithms. As a concrete example, consider a company experimenting with two recommendation algorithms and deciding which one to deploy in production. Symbiosis bias refers to the interference where the performance of one algorithm is influenced by the other algorithm through data feedback loops. Through a bandit formulation, we provide preliminary results on sign preservation properties of such A/B tests.