Well that's the other factor, passing.
Despite how insufferably clever people think it is to go find pictures of post-surgery (and a lot of other work in most cases) trans people and go "Oh so you're saying THIS person should use the mens room? Hardy har LOL knee slap yuk yuk" over and over the Trans closet is even more transparent then the gay closet and that's saying something.
You can "My gender soul is on nuanced complicated spectrum of 147 genders" all you want but 99% of biological males and 99% of biological females are obviously what they are, secondary sex characteristics like height, build, voice, jaw and face structure, hip and shoulder size/shape are a thing. Pure or reasonable levels of androgyny as purely physical state are like not common. Sorry sexual dimorphism in the species Homo Sapien is still mostly a thing most of the time. Take it up with God or Evolution, they didn't ask our opinion.
This ties back into the "We can't set up literal or metaphorical genital checkpoints at the door" thing.
We aren't there yet, won't be on a timeframe worth discussing here, and might never get there on any kind of "We can take 6'4" bearded broad shouldered hairy chested person with a prominent Adam's Apple and a penis and make them into a woman in the sense that they will be perceived to be a woman by 99% of people in 99% of situations" thing. It's not something we can make not true because it would be easier if it wasn't.
"Just don't notice that the person in this gendered space is obviously from the different sex" is not a reasonable standard even assuming everyone is operating with good intentions.