I think it's a matter of body temperature, which obviously is affected by outdoor temperature too. In theory, your internal temperature can still run hot, make you sweat, and then still have trouble releasing the warm sweat if it's too humid.
I'm thinking of my time in skiing, and I recalled seeing steam coming off of my friend on a cold morning. My thinking is that if it was also really humid outside, he might not dry off as quickly.