You're taking too short a look.
Back in the 60s (?) second wave feminists put a fair bit of effort into prying gender roles and gender expression out of sex. The core of the argument was that females are biologically different from males, but that those differences do not and should not be used to prescribe our behaviors, careers, and capabilities. The premise was that females can be doctors and CEOs and politicians just as males can, and that there's nothing inherent in being female that precludes us from participating in society on an equal basis, being paid equivalently for equivalent work, and being recognized as fully developed humans in our own right. Those feminists argued that the set of items that fall under "gender" are those that are socially constructed and fluid. Who should do the dishes, who should wear skirts, who is expected to be obsequious and to give way to the desires of others, who is expected to be quiet and soft-spoken, who is expected to be indecisive, who is expected to focus all of their attention on children and the home, etc.
The intention was a good one, and it was instrumental in the leaps that females made in the 80s and 90s, breaking into politics and business leadership roles, and otherwise shattering that glass ceiling (or at least chipping away substantially).
Then, in the late 90s, early 00s, the activists got hold of that notion and ran with it. They repurposed the word "gender" to no longer refer to socially constructed roles and behavioral expectations... but rather to be synonymous with the relatively new concept of "gender identity". They started their efforts by heavily stressing that sex and gender are different, going so far as to attempt to say that the two are entirely unrelated. This allowed them to take the position that some males are "women" and some females are "men" based on their adopted "gender" regardless of their sex.
Then they shifted the goalposts, and started replacing sex with gender in language, policy, and law... and have now come full circle to squashing the two back together in a really nefarious way.
1) Feminists draw a distinction between gender and sex in order to remove social barriers for females
2) Trans activists expand the separation between gender and sex and argue that gender can be completely independent of sex
3) Trans activists substitute gender for sex in language, policy, and law; at a minimum they add "gender identity" to existing law as being synonymous with sex
What you're arguing for is to regress to step 2 above. I'm more inclined to revisit step 1, and just flat out say that gender is completely irrelevant in language, policy, and law, and that the words "woman" and "man" are understood to mean sex in common usage.