I tend to agree with Sean Connery, but if you give people a license to hit others, they'll use it when they (wrongly) think that they are in the right - as you say.

There are other cases where it is clear-cut. It's always legitimate to hit a woman in self-defense or in retaliation for physical violence.