Just out of curiousity, do you also think that slander should be protected as a constitutional right?
"Congress shall make no law ... abridging the freedom of speech, or of the press..."
Why only mention slander? Fraud, extortion, blackmail, are all also exercises of “Free Speech”. If you truly believe in absolute freedom of speech, without any restriction at all, then you have to believe that I have the right to say to you, “Give me all your money, or else I'll kill you.”
I have to think that the “Your right to swing your fist ends where my nose begins” principle applies here. Any right may need to be suppressed, to some extent, when the exercise of that right becomes a violation of someone else's right. You have a right not to be robbed, so my right to threaten to harm you if you don't give me your money has to be sacrificed in this case.
Even so, I think that in the case of slander and libel, the standard, which I think is generally accepted, and which seems reasonable to me, is that government may not exercise “prior restraint”.
I could publicly accuse you—with no valid basis for doing so—of raping young children, kicking puppies, voting Democratic, or whatever, and nobody can stop me. But if, by doing so, I unjustly cause you harm, then you can sue me for slander/libel. This isn't a denial of my right to free speech, but a way that I can be held accountable for the consequences of abusing this right.
You do not have any right that is violated if I happen to own—or even carry in your presence—a firearm. If I am standing next to you with a gun strapped to my hip, this does not cause you any harm, nor violate any of your rights.