The concept that by granting civil rights to Group A, Group B can claim that - as a direct consequence of that granting of civil rights to Group A - some or all of them in Group B are being disadvantaged (eg by being placed at increased risk to their personal safety).
But surely you aren't saying that granting civil rights to Group A can never imperil the safety, or otherwise disadvantage Group B?
I mean, I assume we have gone over this a lot of times. Let's think about some civil rights.
The right to vote.
The right of free speech.
The right to bear arms.
The right to drive a car.
The right to purchase alcohol.
The right to purchase medicine.
The right to sell medicine at your discretion.
The right to listen to music at a volume of your choice.
The right to fly a drone and take pictures with it.
All of these rights are specifically restricted, frequently. In some cases, quite explicitly because people are part of a Group. (which is to say, a group that is all the people who possess some attribute, such as age, education, criminal records) Moreover if we did not restrict the rights of Group A, Group B would be made less safe.
So, as a general principle, your concept is not something that is relevant.
Which leaves a question of whether, in this specific case (i.e. granting people the right to use the changing room of their choice, would imperil or otherwise disadvantage "Group B", which in this case is females.
I think it would in fact disadvantage them.
In the case of the black/white restroom think, I do not think it would disadvantage "Group B", i.e. white people.