That's what the EA says exclusion should be. 'Legitimate aim' obviously. Whether service providers accommodations are indirect discrimination under the EA may well end up being tested in court if individual providers decide they are not going to make use of the EA exemptions.
Yes, there are certain situations where only exclusion of certain people guarantees fairness, safety, privacy, or dignity, for others.
I'm not allowed in the under 5's ball pool at IKEA. It's detrimental to me but more detrimental to the toddlers I would be landing on. The scales of justice have not been unbalanced by my exclusion.