What's Charles's reputation? I've always considered him a boring old fart, even his scandals are dull stuff. How do others see him? As wickedly malevolent? Sinfully seductive? Mad? Evil? Any of those, alone or in combination, would make him significantly more interesting and therefore maximize his entertainment value, so they'd be quite desirable in a royal...oh, I see! Charles is simply too uninteresting to be a decent monarch! Elizabeth had the war, and her murder spree, and her feuds with the other royals to make her interesting, and of course the silly hats. William has the pretty wife and cute children and his erotic sculpture-carving hobby to make him interesting. What does Charles do? Drink homeopathic tea and write letters complaining about architecture! Maybe, if he's feeling really wild, and has recklessly consumed more than three cups of herbal tea, he'll get out a stamp and collect it.
Screw that. Just give the crown to Harry, who will probably puke into it during a bender, and make the nation all the better for it.