Because men wearing women's clothes (dresses/skirts/high heels) is not considered the norm in our society. It's considered unacceptable for various reasons including linking it to homosexuality, trans-genderism, etc. whether that is accurate or not. For many, that is unacceptable, especially to the conservative minded. On the other hand, it's been culturally accepted and normalized for women to wear men's clothes. Women can wear pants, jeans, t-shirts, shirts...even those actually made for men...and no one thinks twice. The "boyfriend shirt" has become a normal part of a women's wardrobe.