You argument falls down immediately upon examination. If your argument was correct then you would agree that the actions of mass murderers, sexual offenders, thieves, etc, are moral because they are indifferent to the suffering of their victims.
Individuals don't decide morality, societies/populations do. The problem you point out is one that has been addressed many times as in-groups have grown. When clans gave way to tribes and then to cities, nations, and finally empires, moral viewpoints were forced to change.
Instead of me replying to you replying to what you think my argument is to I think Harris's argument is, let me try restarting. But Harris and I would disagree that populations decide morality, at least I think so

.
This is Harris's argument:
Morality and values depend on the existence of conscious minds—and specifically on the fact that such minds can experience various forms of well-being and suffering in this universe. Conscious minds and their states are natural phenomena, fully constrained by the laws of the universe (whatever these turn out to be in the end). Therefore, questions of morality and values must have right and wrong answers that fall within the purview of science (in principle, if not in practice). Consequently, some people and cultures will be right (to a greater or lesser degree), and some will be wrong, with respect to what they deem important in life.
For ease of reading, would rephrase it like this:
1. We are conscious beings, and our mental states can be studied scientifically.
2. The state of well-being is self-evidently valued over the state of suffering.
3. We can use science to determine how to maximize well-being/minimize suffering.
4. Therefore science can determine our moral values, which are objectively and universally right or wrong.
Now, I think he is playing loose and hiding a lot of his arguments. For example, #2 just doesn't cut it. He plays the incredulity card and proposes bizarre hypotheticals to try to get us to admit that we value well-being. The problem is he needs to demonstrate that the value for universal well-being is the sole and overriding value we should have and follow in morality. There is an ought here, not an is. If I am not a boddhisattva that values the suffering of all living creatures equally, he needs to be able to tell me why I should. He can't. He can only point to his maximum suffering or word swapping for health hypothetical "obviousness".
You can see it in his
response to critics (my arguments are down in the Value and Persuasion Problems). He just appeals again and again to "science involves assumptions, so nothing is wrong with me assuming this ought value" and "it is just obvious that we wouldn't say the same about X".
Here is my interpretation of what his argument actually is:
1. We value well-being self-evidently and obviously.
2. We (should) value the well-being of others equally. This is also self-evident and obvious.
3. The value of well-being overrides any other value we may have. Don't ask why or how.
4. We can study well-being well enough scientifically.
5. Therefore we can engage in welfare utilitarianism scientifically.
6. Act/rule? Predictive? Aggregative variety? Motives? What on Earth are you talking about? This is a completely unique system I made up just now and there are not decades of criticism!
6. Therefore all you moral philosophers are wrong, values are facts, there is no is-ought distinction, look at me, look at me!
(If you doubt any of the steps, I will tell you you are self-evidently wrong and are being a stupid-head. Would you rather live in a world of maximum misery? What if I replaced the word "well-being" with "health", huh?)