Trans people are in danger of violence and mistreatment due to how they look; how they act; what kind of genitalia they have. None of these dangers are unique to transpeople.
Indeed, a transwoman who presents as a woman and passes as a woman is in exactly the same danger as other women. Which is kind of the point.
To be fair, transpeople do have a lot more problems with violence. As for people who pass, the problem is not with people who pass. It is people who pass, and then suddenly stop passing. i.e. People assume that they are dealing with a female, and then realize they are dealing with a male. This has been known to lead to violence.
When this issue started coming to the knowledge of society, I had a lot more sympathy for transpeople, because I think casual violence was more prevalent even just 10 years ago than it is today, and I think that an assault on a transwoman for using the women's bathroom would be met with kind of a shrug if it had been reported 10-15 years ago. I think today that a lot more people think it's ok for transwomen to use the ladies' room, and even among people who don't think that's ok, almost everyone recognizes that it is not ok to beat them up if they do use the ladies' room. Law enforcement is not likely to just look the other way as they once did.
Over the years, my attitude towards transgenders has hardened a little bit for a few reasons. The level of violence is less, and the level of entitlement is higher. I don't now and never have had a really big issue with a transwoman being out, looking like a woman, and peeing in a public restroom. However, I do think it's reasonable that in a high school setting, when dealing with people who you interact with every day, to use the faculty restroom when it's offered. When in a locker room, do everything conceivable to "pass" so that you don't make the girls in the locker room uncomfortable and, therefore, in a high school setting where everyone knows everyone, use the private room.
These things are reasonable accommodations, but they go against the trans rights agenda because they all acknowledge that transwomen are not really women. That's anathema to the people who think that they really are women despite their genitals, and there just isn't a way to reconcile the positions. As a result, I've grown more toward the harder line stance.