“If safety was a concern, and there was evidence to select certain bodily characteristics to base safety cut-offs on, then you would see, say, shorter men excluded from competing with taller men, or lighter women from competing with heavier women, across sports.”

Yes, that's correct. In sports where serious physical disparity can easily result in injury, these safety cut-offs exist. Weight classes, age cutoffs, and also sex-divided leagues exist for, among other reasons, this exact issue.