Log In Sign Up

Improved Bilevel Model: Fast and Optimal Algorithm with Theoretical Guarantee

by   Junyi Li, et al.

Due to the hierarchical structure of many machine learning problems, bilevel programming is becoming more and more important recently, however, the complicated correlation between the inner and outer problem makes it extremely challenging to solve. Although several intuitive algorithms based on the automatic differentiation have been proposed and obtained success in some applications, not much attention has been paid to finding the optimal formulation of the bilevel model. Whether there exists a better formulation is still an open problem. In this paper, we propose an improved bilevel model which converges faster and better compared to the current formulation. We provide theoretical guarantee and evaluation results over two tasks: Data Hyper-Cleaning and Hyper Representation Learning. The empirical results show that our model outperforms the current bilevel model with a great margin. This is a concurrent work with <cit.> and we submitted to ICML 2020. Now we put it on the arxiv for record.


page 1

page 2

page 3

page 4


A Fully Single Loop Algorithm for Bilevel Optimization without Hessian Inverse

In this paper, we propose a new Hessian inverse free Fully Single Loop A...

Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey

Representation learning enables us to automatically extract generic feat...

A Constrained Optimization Approach to Bilevel Optimization with Multiple Inner Minima

Bilevel optimization has found extensive applications in modern machine ...

Hierarchical Collaborative Hyper-parameter Tuning

Hyper-parameter Tuning is among the most critical stages in building mac...

AutoCompete: A Framework for Machine Learning Competition

In this paper, we propose AutoCompete, a highly automated machine learni...

Eigendecomposition of Q in Equally Constrained Quadratic Programming

When applying eigenvalue decomposition on the quadratic term matrix in a...

Fair Representation Learning through Implicit Path Alignment

We consider a fair representation learning perspective, where optimal pr...