Kilt, sulu, toga, thawb, kaftan... there have been many variations on garments across the world that aren't pants and aren't dresses. You want to tell this guy that he's wearing a dress?
Dresses are dresses whether worn by men or women. That some don't like to use the word dress is beside the point.
But remember why this came up - it was to do with the legislation that tried to define something as - to paraphrase - "people performing wearing clothes of the opposite sex". The definition was ludicrous and obviously flawed.