You are not giving full details.
Did you serve a party wall notice.Was it valid.Who is your party wall surveyor and did you get a reply from the neighbor.If they dissent as you say, you should have a letter from them appointing this so called Jim as their surveyor.
Confirm this and I can tell you exactly how to get this guy off your back forever....
Hi Bouba,
I served a party wall notice, the neighbour dissented and they "proposed" Jim in the following format:
I am not content for the works set out in your notice to go ahead as proposed and appoint the surveyor detailed below:
Surveyor's name:
Surveyor's contact details: