The Most Rev Justin Welby, who previously served as Bishop of Durham, said it was "absolutely outrageous" to condemn people who raised such concerns as racist.
In an interview with Parliament's The House magazine, he said that the scale of the migrant crisis meant such anxieties were entirely reasonable.
"Fear is a valid emotion at a time of such colossal crisis. This is one of the greatest movements of people in human history. Just enormous. And to be anxious about that is very reasonable," he said.
"There is a tendency to say 'Those people are racist', which is just outrageous, absolutely outrageous."
The Archbishop said it was essential that the "genuine fear" that people felt was listened to and resources put in place to address their concerns.
"In fragile communities particularly - and I've worked in many areas with very fragile communities over my time as a clergyman - there is a genuine fear: what happens about housing? What happens about jobs? What happens about access to health services?" he said.
He said that with the right support, the British people had shown they were able to deal "brilliantly" with the challenges which such situations created.
"It is simply a question of the scale on which we are prepared to act, in a way that spreads the load so it can be managed," he said.
"Fear is justified, I wouldn't want to criticise that for a moment, but so is hope wholly justified, because we have the capacity. We're those kind of people, we always have been."
Iain Duncan Smith, who used to lead the Conservative Party and is now Work and Pensions Secretary, welcomed the Archbishop's comments.
But he warned that, for many years, "the elites" equated anxieties about immigration with racism, allowing groups with "nasty motives" to capitalise on the issue.
He told the BBC Radio 4 Today programme: "I think these are rational comments from the Archbishop and are to be welcomed, but, you know, you wonder just how late they have come from various people in institutions."
He added: "For far too many years, what's happened is that, in a sense, the elites really all said 'It is terrible to talk about immigration and if you do you are racist'.
"So they shut down the debate for many, many years. I can even remember back at the time when Tony Blair was prime minister, to even mention immigration was to be accused of being a racist.
"You talked in terms about asylum seekers, but that accusation probably silenced legitimate discussion and it meant that, if you do that, what happens is you push this debate to the margins, which is what you are seeing in Europe.
"And then political parties and people with very poor intentions and rather nasty motives then start to take this issue, and that's where you lose control.
"So this should have been the case many, many years before but shutting it down has been terrible for the British people."