You may note that I don't consider the feelings of fascists and other societal rejects here. That's intentional.
It's always been ok with them. It's not because of fear of retaliation that you don't see Nazis punched every time they stick their heads out. It's because most people are descent people like you, and they don't think violence solves anything. They were right, too. And then the world changed.
No, I know. I think it's right to punch people that contribute to the gradual undermining of democracy and who spew vile bigotry, trying to move society towards authoritarianism. We've already had this discussion.