算法1 AMSGrad

算法2MAXGrad

输入： $x_1\in \mathcal{F}$ ， ${\left\{{\beta }_{1t}\right\}}_{t=1}^T$ ， ${\beta }_2$ ，

${\left\{{\alpha }_t\right\}}_{t=1}^T$ ， $ϵ$

输入： $x_1\in \mathcal{F}$ ， ${\left\{{\beta }_{1t}\right\}}_{t=1}^T$ ， ${\beta }_2$ ，

${\left\{{\alpha }_t\right\}}_{t=1}^T$ ， $ϵ$

初始化： $m_0=0$ ， $v_0=0$ ， ${\widehat{v}}_0=0$

初始化： $m_0=0$ ， $v_0=0$ ，

1: for $t=1$ to T do

1: for $t=1$ to T do

2: $g_t={\nabla }_x{f}_t\left(x_t\right)$

2: $g_t={\nabla }_x{f}_t\left(x_t\right)$

3: $m_t={\beta }_{1t}{m}_{t-1}+\left(1-{\beta }_{1t}\right)g_t$

3: $m_t={\beta }_{1t}{m}_{t-1}+\left(1-{\beta }_{1t}\right)g_t$

4: $v_t={\beta }_2{v}_t+\left(1-{\beta }_2\right)g_t^2$

5: ${\widehat{v}}_t=\max \left({\widehat{v}}_{t-1},v_t\right)$

and $V_t=\text{diag}\left({\widehat{v}}_t+ϵ\right)$

4： $v_t=\max \left(v_{t-1},{\beta }_2{v}_{t-1}+\left(1-{\beta }_2\right)g_t^2\right)$

and $V_t=\text{diag}\left(v_t+ϵ\right)$

6: $x_{t+1}={\Pi }_{\mathcal{F},\sqrt{V_t}}\left(x_t-{\alpha }_t{V}_t^{-1/2}{m}_t\right)$

5: $x_{t+1}={\Pi }_{\mathcal{F},\sqrt{V_t}}\left(x_t-{\alpha }_t{V}_t^{-1/2}{m}_t\right)$

7: end for

6: end for