Sanekata is referring to a famous earlier poem (SIS XIII: 832) here.

‘Light summer garb’ usuki natu koromo was commonly used in love poetry, both to imply the lightness of a lover’s regard and skin-to-skin contact between lovers.

In his poem, Sanekata says, ‘I know you may not care for me, but what about getting together?’