There are only 2 sexes. The female sex are oppressed because they are female. They are oppressed because of their sexed body. Women cannot identify out of this oppression by claiming to be something else. A girl in Afghanistan cannot say 'I'm a boy, let me go to school'. A girl in Africa cannot say 'Don't perform genital mutilation on me because I'm a boy'.
It's misogyny because it is the oppressor class (men) opting into the oppressed class (women). It is removing women's right to have a unique word to describe themselves. It is removing women's right to single sex spaces that were set up exactly because they are oppressed on the basis of their sex, not how they identify.
Can you understand why people got so upset about Rachel Dolezal, a white woman, pretending to be black for decades? Can you understand why many people would call pretending to be black when you aren't 'racist'?
That's how a lot of women feel about men who identify as women, especially when they displace women in sports, awards, scholarships etc.
Your colleague is no less a male than any other male on the planet. Regardless of hormones, surgery, pieces of paper .... they are, and will remain, male. How you view them is a matter for you, but it does not change reality.
The fact that you say 'Prisons aren't my problem' and 'The sports bodies will have to deal with it' just shows that you don't care about the wider issues. You don't care about how they affect women because it doesn't impinge on your life at all.