I never understood what was so terrible about going in the men's bathroom. Nobody gives anybody any shit in there unless it's like a locker room or something. Normal public bathrooms are not social clubs for men.

I wouldn't kick that poor person out personally but as a matter of principle you're assigned to one, and I'd prefer you use it. If you're not scaring the women it's not even going to be noticed. Lots of women ambiguously look like dudes, sometimes on purpose.