There are the remnants of an old stone wall level with his lawn, but on my side of the boundary.
Do you know who owns the wall? Usually (Often? Sometimes?) when there's a row of adjacent gardens each occupant owns one boundary fence/wall. In which case that wall will be his, or yours.
If it's on your side of the boundary then if it's his you could ask him to remove it, unless he has permission to have it there. But disputes between neighbours over fences, hedges and walls etc can become as intractable as those in the Middle East - you only have to read some of the tales of woe on the gardenlaw forum - so you'd do well to avoid any conflict if you can.
Are you absolutely sure that the wall is entirely on your land? And you can prove it?
I want to put a fence up, replacing the existing one, which was put up by a previous owner of my house on my neighbours side of the boundary, temporarily, before he moved in... which he's complained about.
Fair enough, possibly. Is the fence on his land? If it's still on yours then there's not much he can do about it, particularly if it was there when he bought the house.
But if he doesn't like "your" fence being in his garden, then he ought to understand why you'd like to do something about his wall being in yours, if it is his. If it's yours then you may replace it if you want.
I want to put the new fence on my side of the boundary.
But on which side of the wall? Comes back to exactly where the boundary is. A valid technical and legal solution might be to build a new wall on his side of the existing one, putting in fence posts as you go, then erect the fence and remove the old wall.
Or as a last resort, just put a fence up on your land - there is absolutely nothing he can do about that as long as it's no more than 2m high. Let him deal with the fence already there - you didn't erect it and he knew it was there when he bought the house. If it's on his land he owns it.
https://www.planningportal.gov.uk/permission/commonprojects/fenceswallsgates/
If I remove the wall to dig for posts and his lawn collapses into my side, am I liable for damaging his lawn?
Yes.
I just know I'm going to have problems with him, no matter what.
Thanks.
You really do need to establish exactly where the boundary is, and who is responsible for maintaining any fence or wall, and out of that will come who owns the existing wall and whether it should be on your land, if it is.
Once you've got the facts agreed with him you'll be in a better position to decide what you can/may/should do.
It might turn out to be to your advantage for his garden to have strayed over the boundary, as that would mean you can erect a fence which is 2m high from that, and get you more privacy.