You're both creating a false equivalence.
There is a major difference. The AntiFa hate people because of what they are, the Alt-Right Nazis etc, hate people because of who they are.
Let me explain.
AntiFa are against one particular ideological, Fascism. The claim that they "have no tolerance for ideological diversity or freedom of thought" on totally incorrect. They have no tolerance for those that use that ideological diversity or freedom of thought to push Fascism. AntiFa have no concerns if you want to be mainstream right-wing, libertarian, centre left or whatever, as long as you aren't supporting Fascism. If those supporting Fascism, stop doing so, then AntiFa will leave them alone. If supporters of Fascism all stopped supporting it, or at least pushing for it openly and at the highest levels of Government, AntiFa would collapse and go away too. The things that AntiFa is against is something that people chose to be, it is what they are.
The likes of the Alt-Right, KKK, Neo-Nazis, White Nationalists.... Thy hate people who are different to them, for how they were born. No-one chose to be born a Jew, or Black, or Gay, or Hispanic, but they are targeted for these things despite having no control over it. The Alt-Right truly does not tolerance ideological diversity or freedom of thought, they expect everyone to do and say the things they declare to be acceptable based on their one ideological, and if you don't follow them sand their beliefs, you are a target to be attacked and destroyed. They attack and hate people for who they are, who they were born. As long as there are non-white people, gay people, or people that refuse to support the Alt-Right cause, then they will be hated.
If you can't see the difference in hating a person because of the political ideology they have chosen to push and hating someone because of who they are born, well then you have a problem.