I've been reading a bit of Karl Popper lately, especially his opinions regarding tolerant societies. He argues that if we tolerate the intolerant, we will risk ultimately losing our tolerant society altogether.
Tolerance is not a moral absolute, it is a peace treaty.
Those who do not respect the peace treaty are not protected by it.
Taking "rights" as absolutes ultimately means that no one can have rights, because rights are going to inevitably conflict. "Your right to swing your fist ends at my nose," to quote a libertarian principle. If we cannot set appropriate boundaries for the exercise of rights, addressing the expected points of conflict with reasonable limitations and redress for violations, then we eventually only those strong enough to dominate in a conflict of rights will have them. At that point, they cease to become rights, and end up becoming privileges accorded only to the strongest, Might Makes Right(s).
We've seen this happen far too often. It happened in Nazi German, and it's happened numerous times throughout the US with the suppression of the rights of indigenous peoples, African slaves, and continues to happen to this day.
We already accept a large number of limitations on the exercise of our rights.
For example, free speech.
If there were no limits on where and to what extent this could be exercised, then you'd end up with me standing outside your window shouting at you through a megaphone at 4am. Or while standing in the middle of a city council meeting in session.
If there were no limits on what could be said, I could libel and slander you, ruin your reputation with specious claims and lies, and you would have no recourse.
I could spread conspiracy theories about prominent public figures or personal enemies and demand that they be assassinated.
We've seen what happens when people exercise their "free speech" without reasonable restrictions and limitations. We see bigots use their "free speech" to harass black and hispanic and LGBTQ people; up to and including getting them killed through lying to the police about threats, aka "swatting".
As a society we decide what the limitations on rights need to be to ensure that people are free to live their lives as they see fit, to maintain personal autonomy, while protecting the equal rights of others to do the same. We protect rights up until they interfere with the equal rights of others. Being imperfect humans means that we don't always get it right -- for example, the GW Bush administration's "free speech zone" legislation -- but we shouldn't let the perfect be the enemy of the good. We cannot get it perfect, but we can make changes as needed, as it becomes clear that threats exist and we understand better how to counter them.
Ignoring threats and letting them grow unchecked has never been an adequate counter.