When well-bred men and women wept in Old Japan, they dried their eyes on their sleeves, so ‘wet sleeves’ is a common metaphor for grief. The poet is saying ‘I have sorrow enough for us both’.