My own take is that atheism is a doctrine (as is theism, transubstantiationism, trinitarianism, etc.) and not a religion itself (same as with theism). Discrimination based on doctrine is illegal under the First Amendment.
There can, and are, atheist and Humanist religions.
ESA v. Rylander rejected state arguments that religion must be defined by a belief in a "Supreme Being" and ruled that religious tax exemptions must be granted to the non-theistic Ethical Cultural Society.
Here's a relatively recent federal circuit court decision that says atheism is the equivalent of a religion for purposes of the First Amendment:
http://caselaw.findlaw.com/us-7th-circuit/1467028.html
I think much of this decision echoes what ThePrestige said in post #43, and is a healthy and reasonable attitude.
Getting back to Perry's statement: I think he's trying to contradict
McCollum v. Board of Education which is now very well established law. (That is, it's been relied on for subsequent decisions, and is not likely to be reviewed or reversed.) Government endorsement of a religion (displaying religious icons, prayer or religious services in public school, etc.) violate the Establishment Clause. In that sense, we do indeed have "freedom from religion".
If he only means no one has the right to be insulated from speech acts about religion, that's a pretty silly and trivial observation to make. Did anyone
think we have such a freedom?