The implication in harassment existing as a crime is that you have some other right that's able to at least equal mine. The arguments for any limitation on the First almost always rely on some phantom right that doesn't show up anywhere in the text.

I'm OK with the courts staying out of the legality surrounding "harassment" as long as they're also willing to stay out of the solutions that a person might employ to end "harassment".

In other words, keep it between men. If you cross a line and you get what you deserve, well, you asked for it and I'm personally not going to convict someone for administering justice to you.