Sharp Bias-variance Tradeoffs of Hard Parameter Sharing in High-dimensional Linear Regression

10/22/2020
by   Hongyang R. Zhang, et al.
10

Hard parameter sharing for multi-task learning is widely used in empirical research despite the fact that its generalization properties have not been well established in many cases. This paper studies its generalization properties in a fundamental setting: How does hard parameter sharing work given multiple linear regression tasks? We develop new techniques and establish a number of new results in the high-dimensional setting, where the sample size and feature dimension increase at a fixed ratio. First, we show a sharp bias-variance decomposition of hard parameter sharing, given multiple tasks with the same features. Second, we characterize the asymptotic bias-variance limit for two tasks, even when they have arbitrarily different sample size ratios and covariate shifts. We also demonstrate that these limiting estimates for the empirical loss are incredibly accurate in moderate dimensions. Finally, we explain an intriguing phenomenon where increasing one task's sample size helps another task initially by reducing variance but hurts eventually due to increasing bias. This suggests progressively adding data for optimizing hard parameter sharing, and we validate its efficiency in text classification tasks.

READ FULL TEXT

page 1

page 2

page 3

page 4

05/13/2022

Sharp Asymptotics of Kernel Ridge Regression Beyond the Linear Regime

The generalization performance of kernel ridge regression (KRR) exhibits...
06/15/2022

Noise Covariance Estimation in Multi-Task High-dimensional Linear Models

This paper studies the multi-task high-dimensional linear regression mod...
11/12/2019

Learning Sparse Sharing Architectures for Multiple Tasks

Most existing deep multi-task learning models are based on parameter sha...
07/24/2019

Sample Size Calculations in Simple Linear Regression: Trials and Tribulations

The problem tackled in this paper is the determination of sample size fo...
07/25/2019

Phase Transition Unbiased Estimation in High Dimensional Settings

An important challenge in statistical analysis concerns the control of t...
12/17/2018

Likelihood Ratio Test in Multivariate Linear Regression: from Low to High Dimension

Multivariate linear regressions are widely used statistical tools in man...
10/02/2018

Moderate-Dimensional Inferences on Quadratic Functionals in Ordinary Least Squares

Statistical inferences on quadratic functionals of linear regression par...