You should probably try it some time. It is an interesting experience. It is easy to demonize enemies when they are nameless faceless masses. That changes when you meet them face to face, share a drink, see them smile.
The sport at the Olympics is merely an excuse. An excuse to bring millions of people from all over the world together. Friendships, memories and opinions are forged during those times. Taken back to individual nations and expressed to wider audiences.
The Olympics is about celebrating humanity. It is about bringing us together celebrating what we have in common, and for a few short weeks distracting us from all that divides us.
I already practically live(right along the border) in a very international city. In some areas if I walk just a few miles it is like walking through several different countries - Korea, India, Brazil, Egypt, Greece, Russia, China, etc. And generally, we all get along, the diversity is wonderful. So I don't see how hosting the Olympics could make a place that is already very cosmopolitan even more so. We get tourists and students from just about everywhere.
I am fully sympathetic with your views, although I am not sure if the Olympics do a whole lot to bring about "harmony". It would be interesting to see studies done on this. Indeed, World War II took place only a few years after the 1936 Berlin Olympics. Apparently, the Olympics did not make Hitler and the Nazis recant or at least moderate their racist beliefs(the Olympics similarly did nothing to prevent World War I).
Also, international travel, mass media and the Internet make the Olympics more or less superfluous if their purpose is to bring the world closer together. It's funny when I hear how great the current London Olympics are, due to people all over the world visiting London to participate in the games or cheer on their teams. Yet "people from all over the world coming to London" describes London on any given day of the year. This is certainly a good thing.
I love the idea of the Olympics, but don't like the commercialism or their execution. My main concern is we don't hear enough about the dark side of the Olympics and that not every city that hosts them will benefit. All too often, in the long-term, they don't. The negatives are drowned out by the hype.