Of course the other elephant in the room is the HSE. If they walk on site (and they no longer need to give notice of an inspection these days), and you get caught not using PPE, it gets very expensive. Where I was today there was an H&S notice stating that a walking inspector had come on site atca job in Manchester and caught a joiner using a mitre saw without a vacuum, and not wearing a hard hat, eye protection, ear protection or an appropriate mask. That was 5 infractions, so possibly about £700 to £800 fine for him, plus the site supervisor would have got a fine and the main contractor, too. And you never know which twitching neighbour's curtain might hide a confidential informant (not everybody loves builders)
But hey, fitting in is waaay more important than personal safety any day of the week, isn't it? But then I have the advantage of being old enough that you'd be a brave man to try to tell me not to wear the PPE I deem necessary - I'm also bright enough that if I walk onto a site such as the OP's where H&S is almost non-existent, I tend to just get back in the van smartish and leave. There are plenty of sites where they do want you to go home safe every night, so there is no need to work for unprincipled, ignorant wide boys