To be honest, if I bought a house in that situation I'd assume the fence was the boundary and then do whatever I liked on my side. I don't think the neighbour has done much wrong there. Obviously if they were subsequently rude about it after being told then that is another matter.
They actually dug out the existing concrete boundary posts along with the chainlink fence which was already there and showing boundary clearly, if they didn’t do this there would be no issue.
I could understand if only my fence was there and they thought it was the boundary line?