That's a reasonable point. Clearly a lot of ordinary people do regard harm to other people as acceptable under certain circumstances. What are those circumstances?
In general, it's to avoid maximising harm. In the case of self-defence, there's the feeling that in order to prevent harm to other people, a degree of harm can be inflicted so that suffering can be minimised.
There's also the feeling of justice. If someone is setting out to not consider the harm he might cause to someone else, then it seems fair that he should suffer harm himself - even if this doesn't minimise the total suffering. (Handing over the wallet would be the minimal suffering solution - punching the guy would only be permissable if we think that he had it coming for some reason.
You are missing something important. In my view, punishing a criminal (be that by jail sentence or a punch to the nose) isn't about "justice" or him "deserving it". I don't care about such things. What's important is
incentive. It's important to set a precedent: if you try to rob people, you get punched and/or go to jail. That way, you'll have much less incentive to try it, and the overall suffering is minimized. Thus, by systematically beating up muggers, we can actually lessen the amount of mugging taking place.
The mistake you're making is viewing the situation in isolation. Giving up your wallet may minimize the suffering
for that particular moment, but it will also make the mugger more likely to continue his crimes. By giving up the wallet, you're essentially contributing to the mugging of his next victim. As such, giving the wallet up isn't a moral decision; it's simply a way of keeping your own hands clean. That's the problem with "objective moral rules" such as "it's wrong to hurt people;" too often the rules are applied to isolated situations without thinking about the overall consequences - which are, in the end, the only thing that matters.
There are also many possible ways to extend the view of what is right and wrong. There might be people who would say that if someone wears purple on a Thursday, he should be put to death for offending Shabaz. That's certainly a danger of accepting objective morality. However, under relative morality, it's possible to send twenty million people to a Gulag as class enemies. There are no guarantees.
Of course there aren't any guarantees. Any moral system can turn out to be ineffective or harmful. Which, of course, is precisely the reason why moral systems should be open to scrutiny and change.
Your point seems to be that a relative view on morality is somehow more dangerous than an objective one. I find this baseless, however. It's not like more atrocities have been committed in the name of moral relativism, and most criminals don't subscribe to the idea of moral relativism, or even understand it. Sure, you could theoretically have a moral relativist who murders people indiscriminately. Can you point to one, though?
Anyway, the point of the thread isn't really whether or not objective morality is a good thing, but
whether or not it exists. Can you give me a set of moral rules that is objective, and explain why those rules apply in all situations and all people?
(Also note that "objective morals" doesn't mean "morals accepted by the majority of people" but "morals independent of people and their opinions". Basically, you can't base an objective moral code on opinions, or even outcomes; it has to be based on natural laws or irrefutable logic.)