“I just think it’s important to oppose these people with words, you know, like a real marketplace of ideals,” said the hand-wringing liberal while the neo Nazi nailed a “NO QUEERS HERE” sign outside the marketplace.
The idea is "don't tolerate intolerance." Nazis are definitely intolerant, so it's ok to not tolerate them. Since Nazis promote death squads and gas chambers for people they don't like, punching them is very justified.
There are also some literal Nazis in the United States that are pro-Trump. They are also very punchable.
Does that mean that all trump supporters are punchable? No, it does not. But should it give you pause if you are on the same team as the Nazis? Yes, it should.