You are not required to have a fence.
If you cannot prove who's fence it is then it may well be his, he cannot however force you to pay for one.
If he wants to put up a new fence then its at his cost.
It must however be on his property not yours.
Make sure you mark your boundary line ASAP.
Thanks Matty
The boundary is pretty clear luckily, there are concrete posts set in the ground, he'll never get those out trust me, he can just about manage the fence.
Just had a thought, if he does pull everything down i'll just add another cheap picket fence on our side very close to boundary line.
I'm pretty kind hearted and would normally cave in, but this guy is so petty, amazing how a man could be so selfish at that age.