Do you think it's a good analogy to suggest that giving special privileges to a subset of men on the basis of their feelings is the same as denying equality to whole races of people?
Segregation was a method used by the dominant oppressor group in order to maintain power. Separation in terms of single sex services and spaces was a way to provide safety, dignity, and privacy for an oppressed group (women, including black women), not as a way to further oppress an already disadvantaged class.
Your re-casting women as the privileged class is deeply dishonest. Switching women into being the oppressors by comparing single sex spaces, which exist for very positive & necessary reasons, to racial apartheid/segregation is insulting to everyone who has ever lived under racial apartheid.
Are disabled people also the equivalent of racists for wanting their own toilets?