TBH, I don't see a problem. In fact, those would be more like examples of why it shouldn't be that hard to figure out that something might be non-binary.
For a start, race never was binary to start with, even in the good old racist days. You had black, white, hispanic, asian, etc, all along. That's a whole lot more than "binary".
Second, it was always defined ad hoc, with pretty much no scientific basis. E.g., we call whites "caucasian" because some dude thought that the women from the caucasus were the prettiest. And blacks were for him the bottom of the list because he found them ugly. Then he met a beautiful black woman and changed his mind.
Yep, it's always been THAT much just a case of thinking with the dick instead of it actually being clearly defined different states.
And that's just the tip of the iceberg.
Are the celts (e.g., irish) white or a different race? Well, roll the clock back like a century or so, and you'd find people arguing that they're a different race. No, really, Lovecraft's horrific realization at one point was that he has a very distant celtic ancestor, as opposed to proper pure WASP, so OMG that makes him one of those inferior races.
Some would actually classify them as "coloured", while others ranked them even lower than blacks. In fact there were racist pseudo-scientists classifying them as, I kid you not, the missing link between apes and blacks. (Because, I guess, some paper-white ginger has got to come in between a black chimp and a black human

)
Are Jews a different race? Well, Adolf sure thought so, even while acknowledging in a letter that there is absolutely no genetic base for that.
Etc.
And people being somewhere in between were always a problem for those trying to treat it as a small number of distinct races. That's why you end up with 'theories' like that even a drop of black blood in your ancestry makes you black, even though you might look paper-white to someone who doesn't know about it.
So, yeah, there you go, it was always known that some people are mixed ancestry and thus somewhere in between.
Ditto for ethnicity. How do you even define ethnicity anyway? What you actually have (if you actually study anthropology at all) is some culture, with a number of subcultures, or rather a fluid spectrum of subcultures.
If you go somewhere like, say, Transylvania, you'll find that there is exactly zero biological or other hard difference between an ethnic Romanian, an ethnic Hungarian, or ethnic German, or most of the ethnic 'gypsies' that the former 3 groups look down on. Even language isn't much help there, since a lot are bilingual. All that differs is whether you've grown up in a certain group and identify with them.
And then there's the fact that each of those are a fluid spectrum. You don't even need to move a lot over geography for each of those groups to start looking rather different in customs and whatnot.