It is possible to have a view on immigration while at the same time not being a racist.
However, too often anyone who expresses a view calling for stricter controls is labelled a racist.
As has happened in this thread.
Most of us (and I include both myself and yourself in that, though based on different pieces of evidence) have racist biases and assumptions. Which does make it a bit of a stretch to conclude that a person's views on immigration are ever completely divorced from their views on race.
And, specifically, racists generally do want stricter controls on immigration. If someone who is trying to be non-racist finds themselves in the same camp as pretty well every racist going, they might question whether that is indeed the right camp to be in.
(That's a principle I apply to myself in a different area, where my thoughts on transgender issues put me in the same camp as some pretty nasty people, and I therefore ask myself whether the reasons I articulate to myself for having those thoughts are actually honest and whether that is the right set of thoughts to hold.)