Are you saying that we should throw all the borders open then? If not why not? Does it really matter where you were born? (Not according to you it doesn't).
In an ideal world, yes. However the WHOLE world is pretty big...they literally wouldn't fit.
...however if, we COULD fit the billions from Africa/asia/south America ( which have a much bigger gap in living conditions that the EU countires so therefore would want to move to GB much more) I would have no problem with them competing on a level playing field for jobs.
....remember, it hasn't even been 100 years since we had NO rules stopping ANYONE coming to this country to work, none whatsoever... our country survived like this since ...well forever, let's not think immigration is anything new.