Many libertarians propound what we might call the hardline abolitionist stance on immigration: people have a right to move around; movement is not coercion. Ergo there should be no immigration or emigration restrictions. None, anywhere. No passports, no regulation whatsoever. Full stop.
This analysis is simple enough, but it is also wildly impractical when imagined in our real-world situation, where the state exists and serves as a conduit for forced wealth transfers to all subjects, which includes immigrants. Furthermore, we are rich and the world is poor, so there's plenty of reason to want to move to America.
Surveys that Pew did in Mexico suggest that 40% of Mexicans want to move to the US, if they could... and Mexico is not even a particularly poor country, by world standards. If economic factors drive most immigration, which seems likely to me, then it seems possible that a third or more of the world's population would want to move to America tomorrow, if we abolished immigration restrictions.
How do you think having 2 billion new citizens, most of them desperately poor, uneducated, non-English speaking, not our culture, would affect America? What if it was "only" 1 billion, or even just 500 million?
Well, one thing a lot of libertarians have thought over the years is: well, that would kill the welfare state. Because the tiny minority of native-born citizens, in that scenario, would no longer consent to (or even be able to afford) the level of wealth transfers we'll put up with currently due to our relative uniformity. Uniformity in many ways: wealth, culture (including language), and yes, race. Presumably the native-born would have the power (due to owning most of the wealth) to change the laws. And yes, I am aware that we are not that uniform now: I mean only to say that we are quite uniform now by comparison to how we would be if we abolished all immigration restrictions and a billion peasants immigrated from all corners of the world.
OTOH, there's a second argument that says the newcomers would easily outvote the native-born and they'd vote to dispossess us. Perhaps large parts of our welfare state would go, but not all, and certainly the ideological basis of it would not be destroyed. I subscribe to this position, myself. In this analysis, the fact of the existing state with its mechanisms for forced wealth transfers makes allowing any immigration into a species of coercion; but we can afford quite a bit, and so even the current level of immigration, while perhaps somewhat of a strain, is affordable. But free immigration would not be.
Thus, as an "implementation detail" of libertarianism, it is important that before we abolish immigration restrictions, we abolish the welfare state. Don't hold your breath on that one! Meanwhile, while we wait for the welfare state to collapse, immigration should be limited enough so that we can assimilate our immigrants, economically at least. And given that we do already have laws about immigration, it seems like those would be the place to start. There's something to be said for the rule of law.
Although it is quite possible to graft a racial analysis onto all that, or to emphasize race over culture, language, religion, etc. as a form of diversity, it is not necessary. The simple brute reality of a rich democratic state in a world full of poor people is all you need to force some rather ugly choices. It's not just here, either; it's every Western country.
Another aspect of free immigration that may be dangerous in our current world is letting into the USA vast numbers of immigrants who hate us for our violent interventions in their countries of origin. We are opening ourselves to terrorism if we let just anyone come and go freely. Now, the answer to this is rather like the former problem of the welfare state: abolish it. If we stopped intervening worldwide, gave up on pushing around foreigners, then (after a while) it should be safe to let in all the immigrants we want to, at least in terms of their holding grievances against us. But again, that's not how things are now. So, we do need to become a peaceful country before letting in just anyone.
Unlike the situation with the welfare state, which seems extremely unlikely to change, I think a peaceful USA is an plausible, attainable political goal. It's not only crazy libertarians who talk about abolishing the warfare state and empire: we have friends on the left and right on this issue. But we're not that country now, and so there is a certain prudence in keeping out foreigners who we've offended.
Now, there is one more thing here that many paleocons and paleolibs care about, that I care less about but I still think is worth considering. And that is this whole "national question". If we opened the gates, America would change, drastically. We'd go from being a nation that is post-Protestant, white, and Anglo (with a nice leaven of believing Christians of many flavors, Jews, blacks, hispanics, etc.), to a polyglot patchwork of nations. We would no longer be a nation in any substantial sense, although we would still be living under one state.
Is the demise of American nationhood worth arguing or worrying about? Well, some people really like our country as it is, and I can't blame them for that. I like it too. I don't feel I have the right to coerce people to keep it static, but a lot of people do. Certainly I see no principled reason to abolish immigration restrictionism outside of libertarian thought.
But more importantly, there's a real danger in diversity within a nation-state: once it becomes simply a state, it may not be stable. An unstable state is a dangerous thing, typically full of civil conflict and civil rights violations, sometimes externally aggressive as well. If we look at the world today, we can see many stable nations, and many unstable ones. And the correlation is high between stability and nation-states, as versus unstable states which do not have a (single) nation in their territory. There's also a high correlation between a country's wealth and its level of stability. We may not have the option to be a highly diverse, peaceful, rich country. We may have to choose between being a somewhat diverse, rich nation-state, or a truly diverse poor unstable state at war with itself.
To put some numbers on the value of stability, read this article at Reason. It's a discussion of a 2005 study from the World Bank, "Where is the Wealth of Nations?: Measuring Capital for the 21st Century", which makes estimates of the contribution of natural, produced, and "intangible" capital to the aggregate wealth nations. The findings are interesting: most of the value of living in the first world is not due to natural resources (good farmland, timber, oil, metal ore, etc.), or even already-produced capital (factories, roads, etc). It's "intangible" capital, that is, living under the rule of law in a stable economic system with freedom of contract. If we let in too many people who are not used to the rule of law, who have their own religious law and/or cultural norms, and who have no particular attachment to the common law and inherited Anglo-American national traditions, they may unintentionally destroy our intangible capital.
Now, the preceding analysis is something hard to prove one way or the other, but it seems plausible to me. How much risk should we accept that we'll destroy the rule of law in the USA? It may not be highly likely, but it's a terribly bad outcome.
And, unfortunately, this is an angle on immigration where race and religion, among other forms of diversity, are unavoidable. We are what we are: white English-speaking post-Christians. Diversity, for us, is anything not that, which people notice and care about. To the extent that you buy either of the two arguments about nationality (that you want America to remain more or less as it is, or that you fear that the post-national state will be unstable and thus violent and poor), you will want to exclude immigrants that "aren't like us", in whatever ways seem relevant to you in the above analysis.
Of course, the good liberal response here is that we are not supposed to notice race, religion, culture, etc. Those things are not supposed to matter. Perhaps not, but they manifestly do to most people. We may think all that religion stuff is silly, but ask a believer and he'll tell you that his religion is not silly to him.
My response to the "national question" is as an anarchist: I want to abolish the state. With no state, nations can coexist because there does not have to be one policy. But failing that, I don't think multinational states are stable, at least not liberal democratic ones. (The Ottoman Empire lasted for a very long time, but it is not a model for us.) Thus I think it would be wise, so long as we do have the state, to limit immigration to moderate levels that we can assimilate.