| Method | Network Architecture | |||||
| WRN-40-2 | WRN-40-2 | ResNet56 | ResNet110 | ResNet110 | VGG13 | |
| WRN-16-2 | WRN-40-1 | ResNet20 | ResNet20 | ResNet32 | VGG8 | |
| Teacher | 75.61 | 75.61 | 72.32 | 74.31 | 74.31 | 74.64 |
| Vanilla | 73.26 | 71.98 | 69.06 | 69.06 | 71.14 | 70.36 |
| KD [7] | 74.92 | 73.54 | 70.66 | 70.67 | 73.08 | 72.98 |
| FitNet [8] | 73.58 | 72.24 | 69.21 | 68.99 | 71.06 | 71.02 |
| AT [9] | 74.08 | 72.77 | 70.55 | 70.22 | 72.32 | 71.43 |
| SP | 73.93 | 72.43 | 69.67 | 70.04 | 72.69 | 72.68 |
| CC [16] | 73.56 | 72.21 | 69.63 | 69.48 | 71.48 | 70.71 |
| RKD [17] | 73.55 | 72.22 | 69.61 | 69.25 | 71.82 | 71.48 |
| PKT [18] | 74.54 | 73.45 | 70.34 | 70.25 | 72.61 | 72.88 |
| FSP | 72.91 | NA | 69.95 | 70.11 | 71.89 | 70.20 |
| NST [10] | 73.68 | 72.24 | 69.6 | 69.53 | 71.96 | 71.53 |
| SRKD | 75.83 | 73.94 | 71.84 | 72.08 | 72.37 | 73.36 |