This is always presented as an issue of "trans rights", and who would want to deny "rights" to a small minority in society who are always represented as being particularly vulnerable and marginalised? Women tend to react to a simple question on "trans rights" with a more-or-less unthinking "of course!" If they do think about it, they imagine we're talking about men who are deeply distressed by their sexed bodies, have done everything possible to address this, and simply want to be left alone to pretend to be women without bothering people. Women may feel a little uncomfortable with the idea of such people in our single-sex spaces, but they reason, it's only a very small number of people, how often am I really likely to encounter one, and surely I can put aside my discomfort to accommodate such inoffensive and marginalised people?
When opinion polls have been proposed which spell out exactly what the consequences of self-ID are (any man at all, no requirement for surgery, female-style clothes or even shaving, absolute right of access purely on his own say-so) and that this would inter alia allow any voyeur, flasher, pervert and general creep to self-ID as well, there is usually push-back against this explanation. "How dare you tell people that all transwomen are voyeurs, flashers, perverts and general creeps!" is the usual one. "It doesn't mean that at all!" is another favourite, even though it does.