Perhaps here's a way: the tallness of tall people is the actual characteristic that affects whether they have an advantage or not in basketball, say, and is also the characteristic by which we have classed those people together. But having ovaries (or having had them, in the case of an ovarectomy (?), or having the body for having them, etc., let's not get bogged down in the exceptional cases) is the characteristic by which we class people together into the category of females but the ovaries themselves are *not* the actual characteristic that affects an advantage or disadvantage. If a woman had their ovaries removed, that would not affect their competitive status. If a man, having undergone male puberty, had their testes removed, that would not affect their competitive status (I think).
I have no idea whether this holds up consistently, but I'm sure someone will let me know if it doesn't. But maybe this is a step in the direction of explaining *why* we have different categories of competition for males and females.