Min-Max Multi-objective Bilevel Optimization with Applications in Robust Machine Learning

Abstract

We consider a generic min-max multi-objective bilevel optimization problem with applications in robust machine learning such as representation learning and hyperparameter optimization. We design MORBiT, a novel single-loop gradient descent-ascent bilevel optimization algorithm, to solve the generic problem and present a novel analysis showing that MORBiT converges to the first-order stationary point at a rate of $\widetilde{\mathcal O}(n^{\nicefrac{1}{2}} K^{\nicefrac{-2}{5}})$ for a class of weakly convex problems with $n$ objectives upon $K$ iterations of the algorithm. Our analysis utilizes novel results to handle the non-smooth min-max multi-objective setup and to obtain a sublinear dependence in the number of objectives $n$. Experimental results on robust representation learning and robust hyperparameter optimization showcase (i)~the advantages of considering the min-max multi-objective setup, and (ii)~convergence properties of the proposed MORBiT.