Yes, any argument solely based on discomfort is illegitimate because trust me there are people who dont want to share spaces with black folks, Muslims and gays too and are uncomfortable when they have to. Damn we had men only golf clubs to give a trivial example because the men there weren't comfortable sharing their toys with women, and women were, rightly, up in arms.
Back in high school philosophy class (yeah, it was a thing for me) the teacher taught me about valid arguments and about sound arguments, but never about "legitimate" arguments. I also took Philosophy 101 in college, and the professor affirmed my high school teacher's definitions, and once again left off the description of "legitimate".
Meanwhile, your argument above has the form:
Some arguments that involve comfort levels are not legitimate.
The arguments against allowing biological males into women's only spaces involve comfort levels.
Therefore, the arguments against allowing biological males into women's only spaces are not legitimate.
The Philosophy 101 professor covered that form of argument on day 1, under the general heading of common forms of invalid arguments. Note that it is an invalid argument regardless of whatever the definition of legitimate is.
It seems to me that your declaration of a lot of arguments as illegitimate, whatever that means, is a declaration that you aren't really interested in actual debate. You have already declared the arguments illegitimate, and while you haven't provided a definition of illegitimate, it sounds like it is probably bad.
Now you can argue this is different and there are good reasons for it.... but you have to actually make that argument not just demand everyone agree with you from the get go or they aren't really interested in a debate.
Ok. I will. Let's see if you are actually interested in debate.
Women in our society do not generally take their clothes off in front of men. (The obvious exceptions apply. I assume I don't have to describe them.) They feel discomfort when doing so. Whether that discomfort is a result of societal conditioning or is an instinctive aspect of modern humans is subject to debate, but it is real. From the perspective of the women in the locker room, there is absolutely no difference between a transwoman and a man. They are naked males in the women's only space.
In some cases, people try to equate the discomfort felt in the presence of transwomen to the discomfort than an older generation might have felt to sharing a locker room with black women. This comparison fails for several reasons. First, the discomfort is not caused by being in the presence of a transgender person. It is caused by being in the presence of a male. To say that the people are practicing discrimination against transgenders is to misidentify the discriminant. They are discriminating against men, not against transgenders.
Second, society as a whole has examined the discrimination that affected black people in ages past, and came to the conclusion that there is no difference between black people and white people that ought to justify creating a separate space for the two sorts of people. It was understood that generations of teaching had said that it was inappropriate for blacks and whites to undress together, but that teaching was examined and found to be unsound. The premise that blacks and whites shouldn't share such a space was considered incorrect. Their dark skin really didn't matter, and there was no other discernible difference. In the case of males and females being naked together, we did not reach the same conclusion. The fact that males can impregnate females, or engage in penetrative sexual intercourse with them, was decided to be a big deal. All of the feelings, emotions, and general attitudes related to sexuality were deemed to be significant. Therefore, we continue to segregate males and females. We do so because people are uncomfortable being naked around the opposite sex. Perhaps in some future society it will not be so. I cannot foretell the future, but for now, we have decided it is real.
So, the only question left is whether a person who declares themselves to be a sex (or gender, or whatever) that is different from the one that their biology dictates should be treated in accordance with their self declaration, or in accordance with their objectively measured biological characteristics.
I can continue this later, but I'm sure you can see where this is going, and I must get back to a project. The important part is to note that we do indeed separate men and women. The real question is what criteria we ought to use in order to make that separation. Should it be self-declaration, or biological characteristics. I am prepared to argue, if it is necessary to do so, that biology should be paramount in making that decision.