If they were substantially less fertile than the rest of the Hominidae, they wouldn't have been as successful as they demonstrably were. Do you have any evidence suggesting that Neanderthals were less fertile than Saps?
Not as such. I'm not arguing that they were less fertile, only that this is one of the problems in deciding whether or not viable offspring were possible.
With that said, the only evidence that I am aware of is that they tended to band together in smaller groups. Smaller groups have two obvious potential causes -- lower fertility rates and/or different social structure/interaction. There is no way to tell the difference at this remove, so it's all speculation. It was probably a social thing, though, based on the carrying capacity of the land and their particular diet.
If the cross between a Neanderthal and a Sap is non-fertile, that means that Neanderthals and Saps cannot interbreed. Remember that mules are viable, but sterile. Biologists are comfortable with the idea that breeding that produces sterile offspring isn't "interbreeding."
That is why I said less fertile, not infertile.
Yeah, it's also possile that they all flew away on great big silver space ships, and then only came back to pick up JFK and Elvis. By your own admission, this is an unlikely enough scenario that it can be discounted.
I'm not sure how unlikely it is as a scenario. I don't think it is very likely, but I wouldn't abolutely discount it. Again, those points were not to be taken as an argument for why I think it is likely that humans and Neanderthals could produce viable offspring. They were meant as potential confounders for anyone arguing that it is impossible for humans and Neanderthals ever to have mated successfully given the mitochondrial data. I think it might be possible. But I still think it is very, very unlikely that they could.
As evidence for the above claim (my uncertainty over the fate of any putative "hybrids"), there does seem to have been a bottleneck in human evolutionary history. We are thought to have been reduced to somewhere around 30,000 individuals at some point, were we not? It could possibly (however unlikely) be that any human-Neanderthal hybrids could have perished during that time period leaving behind no progeny.
By the way, JFK is not with Elvis. David Koresh is. They're in a time share in Pago-Pago.