I think the distinction between those who hold extreme viewpoints and those who actively call for violent acts is an important distinction to make. Otherwise, what are we calling "inherently violent"?
Neo-Nazism
is a call to violence, it's inherently violent. Violence against minorities and "undesirables" is a fundamental part of its credo. All you have to do is read the writings of the founders of Nazism to see that. To advocate Nazism is to advocate genocide.
Likewise, to advocate white supremacism is to advocate for the oppression, enslavement, and/or murder of fellow human beings. That's what "white supremacism" means.
I'm not personally comfortable prosecuting and jailing people for "thoughtcrime"; but when neo-Nazis and white supremacists start openly preaching in the streets, openly recruiting on social media sites, openly advocating a worldview that not only invites, but
mandates, violence against anyone unlike them, some sort of forceful response is required in any society that values the liberty, safety, and equality of its members.
At the very least, some form of deplatforming and social sanction needs to take place.
An infamous example was Rwanda before the genocide, where much of the comedy programs on the radio would lampoon Tutsi stereotypes. Even Tutsi would tune in and laugh along. in the end, it was one of the tools to dehumanise them.
I'm reminded of the Minstrelsy period of American entertainment history. The earliest form of distinctly American popular culture, the entire foundation of minstrelsy was constructed of ridicule and dehumanization of African-Americans (first as slaves, then as freedmen). The link between minstrelsy pop-culture, and the treatment of black people in American culture in general, was so closely intertwined that a popular minstrel song gave its name to an entire body of oppressive and racist legislation -- Jim Crow.