This is too abstract and idealized. When you say Group A and Group B, that implies that the differences are trivial. But sex isn't trivial. There are deep differences between them, differences which have no analogue in most other group identities that get discussed, differences that no amount of social engineering can ever paper over. There are situations in which people will respond based on your sex, regardless of your individual behavior. And there are situations in which it's always going to be a hard ask to give someone the benefit of the doubt about how they're going to behave. For example, let's suppose you get arrested, and the cops decide they need to strip search you. That's generally supposed to be done by a member of the same sex as you. The rule isn't that a female can be strip searched by a male cop so long as the male behaves themselves.
Some level of accommodation can be made for trans people, but a transwoman is never going to be treated exactly like a woman, and a transman is never going to be treated exactly like a man. Categories aren't irrelevant. As a practical matter you're never going to evaluate everyone as only an individual. You can't, if for no other reason than you can't get to know everyone you have to interact with.