Then why don't we alter the world to outlaw Nazi/White Supremacist speech entirely? I would think the collective moral judgement is that such speech is morally repugnant.
It's the limited choice landscape again. While I think it is entirely possible to prosecute or try to eliminate certain classes of speech (as has been done with, say child pornography), there is a tension between "oughts" here with no clear winner.
I want to speak freely on sensitive matters. I want to be able to shock sometimes. Choosing to go more toward the censure side means I am agreeing that preventing/prosecuting hate speech has a higher moral value than not. But more than this, the higher moral good should be both clear, obvious and actionable. I don't think, in this case it is.
For example. We could pick the most terrible moral lapse we can imagine, wish whole-heartedly along with all our fellows that such a moral outrage should be guarded against - and still, still, a man might think all about it, dwell on it, and maybe someday act toward this horror. In other words, if I have no ready, workable method to achieve a stronger moral fabric in my society, I'm screwed. Screwed, but freed from the obligation to do anything about it.
So, two reasons. 1) Is it really, clearly, identifiable and actionable? 2) Can it be usefully/practically addressed without ruining other ideas I hold dear?