The free movement of goods is supportive of the economy and so is the free movement of labor. People with certain jobs skills can move where ever their jobs are most valued, without arbitary national lines limiting them.
The issue that people have with immigration is simply someone from a poor country with no job skills is going to move to where ever being in the lowest class of society is treated best. For example someone from the Republic of Congo is more likely to want to move to an EU country than say Brazil, where the poor are still destitute. Because of this countries like the US just give you a green card if you have $1million to invest in US companies, and that's why Asians in the US are generally more wealthy and more well educate because only those with the money to immigrate can do so.
Being on the left generally holds that firstly the pros outweigh the cons in most free immigration situations and also that we totally have the resources to help most of the world out.