I'm pretty sure it's a whole lot less of an issue than you're assuming.
The vast majority of males are visibly male. That doesn't change when they have an orchiectomy and penectomy. At the end of the day, we're a predominantly physically dimorphous species. There are some few people out there that are truly androgynous, and where an average person genuinely can't tell whether they're male or female... but not many. Children pre-puberty can be a challenge, because kids pretty much all look like kids from the outside. At the other end, you get very obese people, where the fat can mask indicators - but I'm not talking about overweight or "technically obese" because BMI calc says so, I'm talking about actually morbidly obese to a point where the fat on their faces masks the facial indicators associated with sex. And even then, there are frequently visual indicators that make it clear 90% of the time - females have breasts that are still identifiably breasts even if they're super fat; males still have facial hair. The number of people who are not morbidly obese adults who are truly androgynous are very few. It can happen, but it's not at all common.
To follow on from that, let's jump in the way back machine and talk about how things worked 20 years ago. One of the lines that gets trotted out is "you've had transsexual males using your spaces forever and you never knew so why should you care now". Well, the reality is that we DID know most of the time. Maybe once in a while we'd get the rare small-statured male with small feet and hands who'd had a LOT of surgery done including facial feminization and tracheal shaving, and we couldn't tell - but that was rare. Most of the time, we could tell that Lumberjack Leanne was probably actually Larry, but we pretended not to notice, and we didn't make a big deal of it.
We pretended because we had confidence that someone with clinical expertise and care had already thoroughly vetted Larry, and at the time, we believed those clinicians had female interests in mind when doing their vetting. We trusted the doctors to look out for us, so any male that ended up in our spaces trying to pass as a woman had already been deemed "safe". And we assumed that they had all had orchiectomies and penectomies. The past couple of decades has demonstrated that we were woefully wrong on both counts. Turns out that over 80% of those historical transsexuals still had completely intact genitals... and even worse, we've learned that the clinicians that we trusted didn't actually give FEMALES any consideration at all when deciding that "this male is allowed to use female spaces". They didn't consult us, and they had only enough care to help train the male how to not make us immediately uncomfortable so that they wouldn't get called out. There was never any actual evaluation for whether the male in question represented a risk or could exploit our good graces.
And over the last 20 years, the entire approach has shifted radically. Now, transgender activists have pushed to remove ANY gatekeeping at all, as well as ANY expectation of surgical transition, and even support the notion that transgender identified males shouldn't even be expected to TRY to pass. Now, the rhetoric is that anyone who say's they're "a woman" has to be treated as if they're a female, and given access without question to female-only spaces on threat of being labeled a bigot - and in some countries prosecuted for hate crimes.
What trust we used to have is irrevocably lost. Where we're at right now is a complete ban to males in female spaces, with no discussion and no exceptions.
+++++++++++++++++++++++++++++++++++++++++++++++++++++
Now, let's take just a moment for some practical reality. Even if the rule is set in stone, and it's no exceptions whatsoever, we're not going to be doing pants-checks at every door. It's impractical and unnecessary. In the rare event that a male sufficiently passes as female (see Blair White for example) that our radar doesn't go off... they're probably not going to be challenged. If they pass enough that we don't suspect that they're male, we're probably going to let it go. But "passing" only works as long as we 1) never see a penis and 2) they don't behave in a way that makes our hackles rise.
