Yeah, I might make a mistake about someone's sex when I meet them in an office or supermarket, but if they take their clothes off, I'll get it right darned near every time, and thinking it's important doesn't make me a bigot.
But that's the linguistic trap we've been lead into by mountains of mush-mouthed non-definitions and constant appeals to "Well it's complicated and personal preference... so it never has to make ANY logical congruent sense."
Nobody (unless maybe Boudicaa if she sticks her head back in the conversation) is going to argue (yet, like I said give it a few years) that preferring a sexual partner to have a specific genital structure makes you a bigot. Nobody is, yet, ready to plant their flag in "You have to like innies and outies the same" territory.
But what they ARE saying and arguing for is a set of definitions and standards for sex and gender where WHY it isn't a problem to pick sexual partners based on genitals can't be answered without directly contradicting yourself. And not in some "HAHA I caught you in a surface level contradiction now I declare you a hypocrite" way like the internet loves, but in actually functionally arguing against your own point on a deep and base directly contradictory level.
Again, and this whole concept just infuriates people for some reason, it's the "You're not saying it, but the arguments you are making is saying it" thing.
Let me put it this way. Ever run into one of the hardcore "Hate the sin, not the sinner" people? They aren't as common as they used to be as people of their stripe have started to own their hatred as lot more in the last few years, but I'm sure they are still around. They were BIG about a decade or two back when gay rights was going through one of its peaks.
Did that argument ever really make sense to anybody? And be honest. Did "Oh I just hate some fundamental part of who you are, but that doesn't mean I hate you?" ever really feel true and honest as an argument? Did you ever encounter that argument and not feel there was no way they were going to maintain, even if they meant it and honestly wanted to, the wall between person and action they claimed they had built in the moral compass?
It's the same thing here. If the entire argument is that anything meaningful about sex and gender is 100% personally dictated BUT you then tack on some "Oh but don't worry that doesn't mean your personal personally personalling along choices about it will EVER be a problem I super-duper pinky swear in fact I'm now going to attack you for even being worried about" it doesn't pass the sniff test.