I think Ali is a good example. We admire him primarily for his athleticism. When we put up a statue of him, we aren't saying "this is a perfect person". We are saying, "That one thing you know about this guy makes him a very special person." We are ignoring that there may be other things about him that make him....not so special.
If we only celebrated perfect people, we could save a lot on bronze, because we wouldn't need any statues.
Would you describe slaveowners and slave traders as persons that aren't/weren't perfect?
You are watering down the hideousness of certain crimes against human beings if you describe the perpetrators as "not so special." You are watering down the hideousness of slaveowners when you compare them to Ali.