Sosei is indulging in a common piece of wordplay here: iro ‘colour’ also meant ‘(sexual) passion’ and so ‘colours fading’ was used to refer to the cooling of interest after an affair.