See, on a basic, physiological level, what is the primary difference in male and female biology? When you get right down to it, it's testosterone.
And the standard low-fat, no-red meat, low cholesterol, plant-based diet that we are all supposedly should be eating to have good health? It's a diet that is primarily instrumental in reducing the bodies production of testosterone.
Saturated fat, and cholesterol are primary building blocks for the bodies production of testosterone.
To put it simply, a low-fat diet, is a low-T diet.
Speaking for myself, It's been 4 years now since I've adopted a high-protein, high-fat diet - I like to call it a Nutrient Dense, Real Food diet, after nearly a decade of low-fat, non-fat, semi-vegetarian, whole grain diet, all through my 20's and early 30's. That diet saw me increasingly getting fatter (at my worst, I was 40 lbs. overweight), and in hindsight, I now realize I had a gradually decreasing libido.
Eating too many plant foods and not enough animal foods was turning me into an Herb.
After a few years of eating paleo, I've discovered that I now have a constant, raging libido of a caveman ready to club the nearest fertile female and drag her away for some carnal savagery. I haven't felt like this since I was 15 years old and first starting dating, and could think of nothing else but sex, drugs, sex, rock-n-roll, and sex. One thing about being older though, while my libido has been rejuvenated by my diet, you don't get the bothersome, can't-control effects like involuntary erections, like you do when you first pass through puberty.