but other workers rent normal flats, and obviously, they will be available to rent before they come, and they will then be available for other people to rent after they leave.
I don't understand your idea that these homes are somehow vacant the day the seasonal workers arrive, but had been unavailable for anyone else to move in. Why didn't your local resident move into them when they were empty?
You said
foreign pickers that come over here for the seasonal work (and take up housing we don't have spare),
How can they "take up" housing that was not vacant?