While I sympathize with those who don't want to "make a fuss," I feel pretty strongly about making a stand as an atheist. As long as we all remain quiet, nobody realizes how many of us there are, or knows that we can be good and decent people, too, or even thinks that their behavior might just possibly be offensive to us.
That stand, however, doesn't have to be rude. I would simply say that I - and possibly other non-xians - don't feel comfortable with the prayer thing and think that the speeches or whatever should be kept religion-free.
I'm reminded of how our really religious friends used to send out super-religious cards at xmas every year. Then one year, we sent out Solstice cards. Those friends were offended. They never thought twice about sending their beliefs to us in the form of a card, but somehow getting ours was traumatic!
The longer we continue not wanting to make a fuss or rock the boat, the longer we will remain an underclass. The only reason we don't have rights is that not enough of us are willing to stand up and demand them.