That's very close to the only sensible explanation I once read, many years ago, when someone similarly bemused brought up this issue in a newsgroup. It's a result of binary thinking, imposed from outside, where everybody who doesn't fit into the standard penis-into-vagina stereotype for whatever reason is lumped together into one "community".
Neither can I, the more so since I think the concept of a "gay community" is a crock of excrement, too, and am unfamiliar with "the gay scene" except for what I've read about it.
Another aspect to my bemusement however is how marginal this issue is, in statistical terms.
To have some hard numbers from an official source, I'll stick to what I can find for Belgium, but I cannot imagine the relative numbers are that different elsewhere in the western world. There is a federal government agency responsible for gender equality, which goes by a traditional Belgian alphabet soup acronym of IGVM/IEFH (they're often that long because they have to be in both Dutch and French). They have transgender issues in their remit as well.
Their website has some numbers (I'm pointing to the French version, hoping that more people might be able to follow that, even though the URL shows the Dutch one was the original.) These numbers don't just include those people who have legally changed, but are an estimate based on how many people have been in contact with various bodies active in the field. They come up with 1 in 13,000 of the male population (i.e. M to F), and 1 in 34,000 of the female population (i.e. F to M).
Translated to the Belgian population of about 10.5 million people, that means there are about 404 M-to-F transsexuals in Belgium, and 154 F-to-M ones, a total of 558. The odds of knowing a member, let alone several members, of such a tiny minority are minute. Even if every one of those people have 100 family members, friends and acquantainces each who know about their situation, and none of those circles overlap, that's still only 55,8000 people with some kind of direct contact, about half a percent of the population. Just how many opportunities of displaying prejudice (or lack thereof) towards transgendered people are there, really?