Selección de variables a través de máquinas de soporte vectorial dispersas : un enfoque de optimización binivel.

Abstract

In this thesis, we address the problem of feature selection through Support Vector Machines with mixed regularization (L2 and L1), called (L2L1−SVM). We present a bilevel approach, where the upper and lower-level objective functions are Mean Square Error (MSE) and (L2L1−SVM) respectively. Furthermore, we use smoothed local approximations of the hinge-loss function and L1 norm, with the purpose of getting a smooth objective function in the lower-level problem and applying algorithms that have been widely studied and improved. This bilevel approach allows selecting the most important features for the binary classification problem and at the same time, optimizing the sparse regularization hyperparameter L1, keeping the regularization hyperparameter L2 fixed. We formulate three variants of the bilevel optimization problem, considering the sparse regularization hyperparameter L1 as a scalar, vector, and grouped vector. To characterize the solution of all formulations of the bilevel optimization problem, we use the Karush-Kunh-Tucker conditions. Based on this characterization, we propose an efficient algorithm called BiSVM L-BFGS to find the numerical solution of the different formulations of the bilevel optimization problem. Experimental results demonstrate the effectiveness of the proposed approach, highlighting improvements in model interpretability and the identification of relevant features.