Sports are a useful wedge issue for transphobes because it's one area where there's probably some legitimate claims being made. Concerns about fairness are always bad faith.
I remain pessimistic that any real solution can be found with the well so thoroughly poisoned by animus though. These discussions about sports fairness do not exist in a vacuum.
I would also point out that for school sports, competition for the sake of competition is not the only concern. Many accommodations are frequently made (and demanded by law) in the name of equity and inclusion. School sports are an important social and educational activity and it's entirely reasonable to make concessions that come at the expense of competition. A high school soccer match isn't actually the FIFA world cup and there are very different concerns and stakeholders involved.
Sports, prisons, battered womens' shelters... These are places where it is obvious that men are not and cannot be women.
Transgenderism as transcending sex segregation is legitimately objectionable, as you acknowledge.
Remove that stipulation, and transgenderism becomes virtually meaningless in public policy. Because gender, divorced from biological sex, is virtually meaningless.
But one won't remove that stipulation, mostly because transgenderism as transsexualism is exactly the TRA agenda. Access to safe spaces for females, by fiat self-ID, is the TRA endgame. So one come up with excuses not to address it. One tries to pretend it's the other side's fault that one can't come up with any solution. One tells oneself and the rest of us that it's our bad attitude that gets under one's skin and prevents one from doing what one knows is the right thing to do.
But also one won't remove the transsexual stipulation because without that stipulation, there's no work left to do in public policy. It is already established as a matter of law that people can't be discriminated against on account of their "gender". A dude wearing a frock is already entitled by law to all the same privileges in housing and employment and elsewhere as a dude wearing a pair of pantaloons joined by a codpiece.
Take away the issue of sex segregation, and the matter of transgender rights is already solved in public policy.
Tell me I'm wrong. Give me an example of a transgender discrimination that is unjust, permitted by law, but not actually an example of transsexual discrimination. If you can find one, I will join you in advocating to have it addressed.