Look at it this way. Let's look at once common but slightly older and archiac term that has mostly fallen out of use... crossdresser. A crossdresser was (is, I'm sure the subculture still exists in some context) a person who preferred to wear the clothing of the other gender.
See the problem? For that concept to have any meaning at all you have to agree that men and women are required by some level; social, legal, whatever to wear different clothing. If you don't, if you think men should be able to wear skirts and women to wear to wear pants and so forth... the concept stops being meaningful.