You didn't say "that's personally not my bag," though, did you. You insinuated, nay CLAIMED, that such people are fundamentally damaged or wrong.
The important question is: should they be spanked for their deviant thoughts?
Concerning fantasies: There is a difference between having a fantasy, enacting a fantasy in a controlled environment with a consenting participant*, and committing crimes in order to satisfy a desire.
*a certain amount of suspension of disbelief is needed, which puts further distance between the fantasy and real life situation.
I doubt people who get off on having their partner wear a French maid costume, expect them to clean the entire house for minimum wage before they get to the naughty part.
Pedophiles can have fantasies, and if they do not involve other people in them, we might never know.
They can't act out their fantasy with a consenting person, so any action towards fulfilling their sexual desire would be criminal.
Someone who is attracted to children and places him- or herself in a position where they can expect to be around children (unsupervised) or have authority over children, is acting in bad faith.
I have no data on the topic, but I would say that any adult who is sexually attracted to the children they teach/supervise/care for should be removed from that position, because they are a risk.
Then there are those that have actually abused children. Some of them are pedophiles, some of them are opportunists who just get off on sexual assault, and children are usually an easier target than adults.
Whatever fantasies these people have or have had is irrelevant, because we are punishing them for their actions, not their thoughts.
None of us are mind readers and people who fantasize about raping little kids are really really unlikely to admit to that, so any attempt to use someone's fantasies as apredictor of the likelyhood that they will become rapists will be dommed to fail.