I'd say your theory is spot-on.

The one time in my office we had a temp-worker come in whom was unobjectively attractive, and ever other woman in the office lost their goddamn minds.

Moreso than normal, that is.

The kicker was, she wasn't even that pretty. She looked good, yeah, and could actually rock it with skin-tight jeans, but she wasn't going for obvious sex appeal or the like. She just looked good and dressed well, and all the other older and/or fat women couldn't fucking stand it.