The thing is, historically it's true that the UK has been colonised by immigrants, and if he'd left it there it would have been hard to argue. That's not what he means, though.