Method

Vanilla

KD

KD+ SRKD

AT

AT+ SRKD

SP

SP+ SRKD

Teacher

T: Resnet20

Top-1

78.25

86.52

86.81

86.77

87

86.5

86.74

85.81

S: Resnet8

Top-5

98.69

99.25

99.4

99.31

99.42

99.14

99.38

99.22

T: Resnet32

Top-1

85.81

88.67

88.97

88.83

89.06

89.05

89.09

86.97

S: Resnet20

Top-5

99.22

99.41

99.49

99.48

99.51

99.52

99.53

99.37