I agree with this, and I think it's a real problem in some cases.
Earlier, I referred to the Charlottesville protestors as Nazis, although I noted that they might not call themselves Nazis. In that case, though, I didn't feel very guilty about it. Some of the protestors carried flags which, although they did not have swastikas, had black and white artwork that was clearly reminiscent of Nazi era designs. Others chanted an actual Nazi slogan, "Blood and Soil". Therefore, I didn't feel too bad about simplifying the discussion by calling them "Nazis".
On the other hand, there are an awful lot of people who express a certain low level of racism in their actions (e.g. by moving away from schools where their kids might have to attend with black kids), and a fair number of people who openly express some sort of racist ideas. It would be inaccurate to call these people "Nazis" or "white supremacists", and often the distinction is important. The most obvious effect of that conflation is to alienate fellow citizens, and voters, and possibly push them into the other camp. If you call everyone who agrees with anything that the Proud Boys ever say a "white supremacist", some of those so named will respond with some variation of "If this be white supremacy, then let's make the most of it."