I think it was writer Lillian Hellman who once observed, speaking about Warhol transgender film star Candy Darling, "Darling is not a woman. In order to be a woman one must first grow up as a little girl."
There are many physiological differences between men and women. It's a lot more than having "female brains." Here's a site that lists most of them.
Link
A lot of the things at that link are either averages (my wife must not be female because she's 70", the same height as the average man there) or are secondary sexual characteristics that can be affected by hormones, such as muscle mass. There are lots of better examples, such as chromosomes, DNA, etc.
But the underlying issue is: That's great if everything that we can identify about a person is all one sex or the other. What about physically intersexed people, easily identifiable by the methods we have now? If such a person is raised one sex but insists on identifying as the other and switches, is the way they were raised the determining factor? How can one tell each of these people whether they're male or female?
http://www.isna.org/faq/conditions
In fact, I think we're at an uncomfortable point in medical knowledge, where we can at least say that intersex people are intersex, even if we can't be confident what gender to assign them. But transexuals cannot be physically identified, so they are left trying to communicate their condition.
If an intersex person says they are unhappy with the gender assigned at birth, they can self-identify as the other and be supported with treatment. So apparently doctors cannot predict which physical component overrules the gender of the brain.
From
http://www.isna.org/compare (bold added)
"The doctors and parents recognize, however, that gender assignment of infants with intersex conditions as male or female, as with assignment of any infant, is preliminary. Any child may decide later in life to change their gender assignment; but children with intersex conditions have significantly higher rates of gender transition than the general population, with or without treatment. That is a crucial reason why medically unnecessary surgeries should not be done without the patient’s consent;
the child with an intersex condition may later want genitals (either the ones they were born with or surgically constructed anatomy) different than what the doctors would have chosen."
I don't know how much problem they run into being told they''re not
real men or women afterwards. I suspect it's less than transexuals, but I don't know. If it is less, that's one reason it would be helpful to identify that brain anomalie that causes a person to feel transexual, and then they would be in the intersex category, rather than being identified by many people as someone who was really born all one sex or the other, or someone who just wants to change sex on a whim.