sex might as well be a binary, because the overwhelming majority of people conform to the definitions of either male or female, which involve differential gamete production (sperm vs. eggs), and only slightly fewer fail to conform to a binary of other primary sexual characteristics (appearance of genitalia) or secondary sexual characteristics that appear at puberty (breasts, pubic hair, etc.).
To be a bit more precise, biological sex in humans is bimodal: if you do a frequency plot with “sex” on the X axis and “frequency of individuals conforming to that sex” on the Y axis, you get a huge peak at “male”, another huge peak at “female”, and then a few tiny blips in between that conform to hermaphrodites or intersexes.
There’s a reason why sex is a binary: evolution produces two distinct sexes who mate with each other to produce offspring. Exactly why there is sex rather than all of us budding off clones or reproducing in other asexual ways is an unsolved problem, but once there is sexual reproduction, you can construct a reasonable theory about why there should be two of them, and that they should be distinct. (A few species have “mating types” that encompass more “sexes”, but these are virtually nonexistent in vertebrates.)