I think this argument confuses necessary and sufficient conditions. There are ordinary, common, karyotypes, and those often go with certain configurations of genitalia, but there are exceptions. So, I think your conclusion is that "gender must be defined by something else, right?" As if somehow the existence of the exceptions make karyotypes and genitals irrelevant to the classification.
I have said it in the past, but I'll repeat. If person A and person B are capable of engaging in sexual intercourse with the result that person B becomes pregnant, then person A is a male and person B is a female. There are no exceptions. It is a sufficient condition for determining sex.
From there, you can pretty much work out almost everyone else. There are a few exceptions, but not many.
Of course, that is talking about the male or female sex. If, instead, you want to talk about a male or female gender, it seems to me that it could mean whatever you want it to mean. It used to mean "sex", but now we're working on finding a new definitions.