Not a single man on this thread has offered a solution that involves anything other than women having to give up something. Including you. Transmen using men's single sex spaces is a different issue - they are unlikely to be a safeguarding issue to men. But if men wish to exclude them, that's up to men. They should have every right to do so.
I'd rather think that you were trying to be kind and inclusive than think it's just that you don't give a toss about women.
Have you ever been on the receiving end of a "joke" about how parts of your body looks, compared to how they feel it should look, from as it turned out a trans man*. The standard answer from me, "it's what cancer left me with", got an immediate answer/response of "sorry to hear that, didn't realise and I shouldn't have said it".
And if we wished to exclude them, we'd have people, like yourself, telling us we were overstepping an unseen line and we weren't thinking about the effect of what we were saying.
As for no safeguarding problems with women in men only spaces, female staff entering with a mobile phone f'instance, it's not just men, but boys as well.
Are you happy that your male kid(s) might be caught on camera in a changing area?
*I found that bit out in the cafe where he insisted we go, as he was buying to say sorry for what he'd said.