Well, I think the first thing to do is to attempt to define what "evil" and its opposite "good" are. I think of them both, not as immovable constants, but as reference points on a sliding scale. My own opinion is that "good" refers to those things, behaviors and actions that respect and promote human life, human liberty, and human happiness as well as those things that increase the ability of many humans to peacefully coexist. And evil refers to those things, behaviors and actions that limit or reduce human life, human liberties, or human happiness, as well as those things that decrease our ability to peacefully coexist.
I also hold as a basic premise of my moral code that all humans have a fundamental right to their own life, liberty and happiness, so actions that, all other factors being equal, increase my happiness by equally decreasing your happiness, (like me stealing your money, for example) are not neutral, but actually usually sit on the "evil" side of the spectrum.
That said, I believe slavery sits on the "very evil" side of the spectrum, because it removes all the human fundamental rights; it reduces human life, it reduces human liberties, it decreases human happiness, and it reduces our ability to peacefully coexist.