It's also a bigger deal for the defendant, who in many cases will be wealthy and powerful and politically connected, while the plaintiff is none of those things. A paper trail is hard to establish in most cases, and a "gentleman's agreement" of like minds leaves none.
As for why slander litigation doesn't occur, there may be many reasons, some good, like a cost benefit analysis that says it's cheaper and less harmful to ignore it, especially if it's spurious or comes from a disreputable source; and some not so good, like the realization, especially in this age when "fake news" seems a passable defense, it's less dangerous to engage in a bickering match than in a trial where truth might actually be presented.
Either way, I think the lack of lawsuits has little relevance to whether the charges are true.