STATS 606: Computation and Optimization Methods in Statistics
University of Michigan
including slides by Stephen Boyd and Lieven Vandenberghe
Given $X\in\reals^{n\times p}$ and $y\in\reals^n$ $(p > n)$, find the sparsest $\beta\in\reals^p$ such that $X\beta=y$:
$$ \begin{aligned} &\min\nolimits_{\beta\in\reals^p} &&\textstyle\|\beta\|_0\triangleq\sum_{j=1}^p\ones\{\beta_j\ne 0\}\\ &\subjectto && X\beta = y \end{aligned}. $$The $\ell_0$ "norm" is hard to minimize (it's not even continuous), so we relax the problem by replacing the $\ell_0$ "norm" with the $\ell_1$ norm:
$$ \begin{aligned} &\min\nolimits_{\beta\in\reals^p} &&\|\beta\|_{\color{red}1}\\ &\subjectto && X\beta = y \end{aligned}. $$Given $X\in\reals^{n\times p}$ and $y\in\reals^n$ $(p > n)$, find the *sparsest* $\beta\in\reals^p$ such that $X\beta \approx y$:
$$ \begin{aligned} &\min\nolimits_{\beta\in\reals^p} &&\textstyle\|\beta\|_1\\ &\subjectto && \|X^\top(X\beta-y)\|_\infty \le \lambda \end{aligned}. $$The Dantzig selector can be formulated as an LP:
$$ \begin{aligned} &\left\{\begin{aligned} &\min\nolimits_{\beta\in\reals^p} &&\|\beta\|_1\\ &\subjectto && \|X^\top(X\beta - y)\|_\infty = y \end{aligned}\right\} \\ &\quad\equiv \left\{\begin{aligned} &\min\nolimits_{\beta_+,\beta_-\in\reals^p} &&1_p^\top(\beta_+ + \beta_-)\\ &\subjectto && X^\top(X(\beta_+ - \beta_-) - y) \preceq \lambda\ones_n \\ & && X^\top(X(\beta_+ - \beta_-) - y) \succeq -\lambda\ones_n \\ & && \beta_+,\beta_-\in\reals_+^p \end{aligned}\right\}. \end{aligned} $$Given $X\in\reals^{n\times p}$, $y\in\reals^n$, find a sparse $\beta\in\reals^p$ such that $X\beta \approx y$.
Basis pursuit denoising (BPDN):
$$ \begin{aligned} &\min\nolimits_{\beta\in\reals^p} &&\|\beta\|_1 \\ &\subjectto &&\textstyle\frac12\|y - X\beta\|_2^2\le \sigma^2 \end{aligned}. $$LASSO:
$$ \begin{aligned} &\min\nolimits_{\beta\in\reals^p} &&\textstyle\frac12\|y-X\beta\|_2^2 \\ &\subjectto && \|\beta\|_1\le\rho \end{aligned}. $$Lagrangian BPDN/LASSO:
$$ \textstyle\min_{\beta\in\reals^p} \frac12\|y-X\beta\|_2 + \lambda\|\beta\|_1 $$