This thread seems to confuse knowing how to use various shortcuts to do arithmetic with the ability to be proficient at math. To me they are totally different.
If you think that, then I guess I am failing to make my point. I saw a couple of examples of "shortcuts" that I know to be occasionally taught. My ex-wife showed me a bunch of "shortcuts" she learned in a college course she took on teaching math. They were new to her as well. She only taught a few when she was a teacher. Other examples in this thread were not shortcuts but rather examples of different mental processes for attacking the problem. These are not taught - it's just how people think.
My point is that natural aptitude in math runs the gamut from those who will never grasp basic math no matter what to the Human Computer Guy (Danny T). Some people can visually do math the "long way" in their heads; others cannot. Some learn shortcuts. Some of us attack problems in our own ways that are efficient and comfortable for us.
I believe that answers the OP, which asked why "so many folks, including highly skilled individuals in their fields, fail so miserably." There's no reason to expect everyone to be good or even comfortable with math just as there is no reason to expect everyone to be able to draw a cat's face, which is literally child's play - hang out in art class for first graders to see what I mean.
I believe that many with a natural aptitude for math don't realize that they even have such an aptitude. They think it's like that for everybody. It's not. And while you can teach certain rote methods, for many people those rote methods are just that. They never "get it" so they are never comfortable with it.
Consider this example given earlier:
47 X 53 = ?
92 X 88 = ?
76 X 84 = ?
I looked at that for a few seconds and quickly realized the "shortcut" that was implied. I had never heard of it before. Anybody could be taught to recognize such a problem and to use the shortcut, but I argue that if you don't realize it yourself without being told, you'll never "own" it.
People without the right aptitude have to look at each problem and run through a mental checklist of available shortcuts they have been taught. It's like a toddler using a shape shorter - the child will try the shape in each hole until it fits. A developmental milestone is met when the child recognizes the shape in his hand and associates it with the corresponding hole. Some people never make the connections that come naturally to others.
Everyone's aptitude has its limits. I remember in junior high school geometry we did proofs. I found them incredibly easy and enjoyable. Other very bright students were stumped, which I found puzzling. I also remember learning about parabolas and such. It wasn't too long before I could see in my head what it would look like without having to make the graph. As I recall, very few of us could do that. Even students who got better grades than me (I was lazy and didn't check my work) couldn't do it and commented that it was "weird" that I did.
My education and career path never required calculus, but my ex-wife's did. I remember helping her with it. I never actually did any of the math, but I could I explain to her in principle what was happening and what needed to be done. She got good grades, but I don't think she ever "owned" it.
In fact she was one of those people who
hated being asked to do math in her head. She didn't really like doing it on paper. She worked hard in school and graduated college with some level of cum laude that I forget now. She was one of a handful of teachers offered an "open contract" before graduation, which in effect means, "We want you to work for us. Sign this and we'll find a spot." She was a successful student who put in lots of effort.
And yet I think some would consider her a "math averse" type of person because she would balk at having to do simple math in her head. Meanwhile, someone like myself with much less training in math and who really put very little effort into it would be considered the opposite.
If you want to "judge" us on our dedication, she would win hands down. Only nobody ever judges me because it comes easy to me. Those who judge her would come to the wrong conclusions unless they knew her like I did.