I've been reading a bit of Karl Popper lately, especially his opinions regarding tolerant societies. He argues that if we tolerate the intolerant, we will risk ultimately losing our tolerant society altogether.
Popper said
"Unlimited tolerance must lead to the disappearance of tolerance. If we extend unlimited tolerance even to those who are intolerant, if we are not prepared to defend a tolerant society against the onslaught of the intolerant, then the tolerant will be destroyed, and tolerance with them. — In this formulation, I do not imply, for instance, that we should always suppress the utterance of intolerant philosophies; as long as we can counter them by rational argument and keep them in check by public opinion, suppression would certainly be unwise. But we should claim the right to suppress them if necessary even by force; for it may easily turn out that they are not prepared to meet us on the level of rational argument, but begin by denouncing all argument; they may forbid their followers to listen to rational argument, because it is deceptive, and teach them to answer arguments by the use of their fists or pistols. We should therefore claim, in the name of tolerance, the right not to tolerate the intolerant."
The same could be argued about free speech. If a society allows unlimited, absolute free speech to everyone, then by default, it must extend that allowance to those who would ultimately wish to remove the right to free speech, and if that happens, society risks losing that right altogether.
This is what happened in Europe in the 1920's: Nazi's came to power from very small beginnings, from discussions in drinking establishments and poorly attended public meetings. Groups such as Freier Arbeiterausschuss für einen guten Frieden, the Free Workers' Committee for a Good Peace) and Deutsche Arbeiterpartei (German Worker's Party) formed. Very minor, very low key, just like some of the extreme right wing groups we have today. These groups grew and grew, and after a few setbacks, ended up coming to power, and the result was total suppression of opposing views, and millions of people killed in ten years of Holocaust and five years of global warfare.
Americans probably see modern Nazi's as not unlike The Blues Brothers "Illinois Nazis", a bunch of incompetent, unintelligent man-children playing at being bad guys. I promise you, those in Europe have a very different view - they see the far right as a real threat, because they are.
In France, Marine Le Pen leads the National Front, a far right idealogical party that, under her, has distanced itself from the anti-Semitism of her father, and has instead, taken up opposition to immigration from the Middle East - they have simply replaced the Jew Boogeyman with the Muslim Boogeyman. The National Front is a neo-Nazi organisation in every way but name, and in the 2011 elections, they were the third most popular party - that's about the same level of support the Nazis had in the early 1930's Weimar Republic.
Do not dismiss neo-Nazis lightly!