Izumi Shikibu is using a reference to another poem to add to the sense of her own here: uFage are actually the feathers on a waterbird’s back, not down, so I have used a bit of poetic license in the translation.
Izumi Shikibu is saying, “I remained faithful last night, but I suspect you did not!”