算法1：蒙特卡洛Q-learning算法

1) 初始化矩阵使 $Q_2\left(0\right)=I_d$

2) for $t=0,T$ :

3) $Q_2\left(t+1\right)=Q_2\left(t\right)+\alpha \left[\frac{1}{s}{\displaystyle \sum }_{k=1}^s\left[N_k+\gamma {\Lambda }_k^{\text{T}}\Pi \left(Q_2\left(t\right)\right){\Lambda }_k\right]-Q_2\left(t\right)\right]$

4) $K_0\left(t+1\right)=\frac{1}{1-\gamma }{\Gamma }_0\left(Q_2\left(t+1\right),0\right)$

5) $t=t+1$

6) end for