What's your position on trans men in the women's toilet? E.g., what if they were wearing really good "manface" but needed to use the tampon machine?
Transgender identified females are female. Just as with the vast majority of transgender identified males - they don't pass as well IRL as they do in carefully planned photo shoots. Most transgender identified females don't clock as males unless they grow a full beard - and even then, it rarely lasts through a second glance.
That said, I have two overarching comments on this.
First - it's easier for females to pass as male than the other way around. Partly this is because the secondary characteristics of males are easily visible, and we've evolved to see them - and to classify anyone who has them as male. Those secondary sex characteristics are prompted by testosterone, and testosterone is a one way street in humans - once it kicks of a particular physiological change, cessation of testosterone doesn't make it unwind. So when a female takes T, their voice box will thicken and enlarge and their voice will drop, they'll grow facial and body hair. When they stop taking T, their voice box doesn't slim back down, and their facial hair doesn't go away. Testosterone prompts permanent changes.
On the other hand, the changes prompted by estrogen are temporary. Most of the changes that females experience during puberty are related to a cycle of several hormones, and some of them are direct changes to reproductive anatomy - enlarging of the uterus, menarche, widening of the pelvic aperture, maturation of lactation glands, etc. You have to have female anatomy for that to happen. When a male takes exogenous estrogen, their skin gets softer and smoother, and their body stores fat in the breast region, as well as in the buttocks and hips. When they stop taking estrogen, however, their body reverts to its natural state because those effects are not permanent physiological changes.
Second - it's a question of who is taking the risk. When a female decides that they want to use male facilities, they are voluntarily taking risk on themselves. That female does not represent an increased risk to any of the males present, the likelihood of a female being able to overpower and assault a random male is extremely low. When a male decides they want to use female facilities, they are placing risk on all of the females in that facility while not taking on any additional risk themselves.
If a chicken decides it's a fox, and wanders into the fox's den... it's the chicken that is at risk, not the fox. When a fox decides it's a chicken and wanders into the chicken coop...
it's still the chicken that is at risk, not the fox.