Tengyuan Liang is a Professor of Econometrics and Statistics at the University of Chicago, Booth School of Business. His research uses principles from Learning and Statistics to understand models and data. His research is supported by the NSF CAREER grant and the William Ladany faculty fellowship. His current research aims to: bridge the empirical and theoretical gap in modern statistical learning; understand optimization and inference of infinite-dimensional models; explore the role of stochasticity in solving non-convex optimization.
Title: Blessings and Curses of Covariate Shifts: Adversarial Dynamics, Directional Convergence, and Equilibria
Abstract: Covariate distribution shifts and adversarial perturbations present robustness challenges to the conventional statistical learning framework: mild inconceivable shifts in the test covariate distribution can significantly affect the performance of the statistical model learned based on the training distribution. The model performance typically deteriorates when extrapolation happens: namely, covariates shift to a region where the training distribution is scarce, and naturally, the learned model has little information. For robustness and regularization considerations, adversarial perturbation techniques are proposed as a remedy; however, careful study needs to be carried out about what extrapolation region adversarial covariate shift will focus on, given a learned model. This paper precisely characterizes the extrapolation region, examining both regression and classification in an infinite-dimensional setting. We study the implications of adversarial covariate shifts to subsequent learning of the equilibrium—the Bayes optimal model—in a sequential game framework. We exploit the dynamics of the adversarial learning game and reveal the curious effects of the covariate shift to equilibrium learning and experimental design. In particular, we establish two directional convergence results that exhibit distinctive phenomena: (1) a blessing in regression, the adversarial covariate shifts in an exponential rate to an optimal experimental design for rapid subsequent learning, (2) a curse in classification, the adversarial covariate shifts in a subquadratic rate fast to the hardest experimental design trapping subsequent learning.