Subgradients

STATS 606: Computation and Optimization Methods in Statistics

University of Michigan

including slides from Stanford's EE364b

Minimization

Recall

$$ f(x_1) \triangleq \min\nolimits_{x_2}F(x_1,x_2) $$

is convex when $F$ is (jointly) convex (in $(x_1,x_2)$).

To obtain $g_1\in\partial f(x_1)$

  1. find $x_2^\star\in\argmin_{x_2}F(x_1,x_2)$ (so that $f(x_1) = F(x_1,x_2^\star)$)

  2. find $(g_1,0)\in\partial F(x_1,x_2^\star)$

Minimization

$$ f(x_1) \triangleq \min\nolimits_{x_2}F(x_1,x_2) $$

Pf: any $(y_1,y_2)$ and any $(g_1,0)\in\partial F(x_1,x_2^\star)$, we have

$$ \begin{aligned} F(y_1,y_2) &\ge F(x_1,x_2^\star) + \begin{bmatrix}g_1 \\ 0\end{bmatrix}^\top\begin{bmatrix}y_1 - x_1\\ y_2 - x_2\end{bmatrix} \\ &= f(x_2) + g_1^\top(y_1-x_1). \end{aligned} $$

Minimizing (both sides) with respect to $y_2$, we obtain

$$ f(y_1) \ge f(x_1) + z_1^\top(y_1-x_1). $$