Did you miss the bit about 13 million displaced, 7 million refugees, and just 10,000 in the UK?
Why? Why not ask them, rather than make assumptions. Some have family legally living here already. Some speak English. Some attended English universities. Some work for British multinational companies. In short, they all have a tie to the UK that makes it more sensible for them than any of the other countries, which is why they are making the effort to get here.
Let's think about our big crisis of the 20th Century - Hitler. If Hitler conquered Europe and you didn't want to become a Nazi, and were worried they would here what you said about them and string you up, you might think about fleeing for your life. You'd probably look to America or Canada, being English speaking countries where many Brits have already gone. But based on your argument, you'd have to go to Morocco, that being the nearest non-European country. Although Germany was also trying to conquer North Africa, so you might need to settle with Chad. Where would you take your family?