A fundamental part of the Harry Potter series was that the villains claims and motivations were based in large part on promoting hatred and violence against "inferior" people - muggles and non-humans. Its a very clear parallel to real-world groups that have derived power from stirring up hated and animosity against a group of "others". There's also strong links to policies associated with such hate groups such as repressing education and controlling the media narrative, and a bureaucracy that refuses to acknowledge that problems exist. People tended to assume that since Rowling portrayed such attitudes as villainous, she would find the overall concept to be villainous, not simply based on which subgroup you happened to be attacking.
Every group that attacks people based on their identity has always claimed its based on legitimate reasons. The blacks are godless savages. The Jews have secret plans to control the world. The homosexuals are child molesters. Transgender people just secretly want to assault women. Its always claimed to be about safety or defense and its always, always been a load of bull trying to cover the same premise of "make an enemy for people to hate to get them to support you".