mykp said:
Does anyone know the legal positions on this?
In practise, if he sued you for the payment, then you would make a counter-claim for compensation, and the court would decide on a fair compromise.
Pragmatically, you can make the builder an offer, pointing out that you are reducing your payment to him by an amount that you would otherwise seek to recover at court. It is up to him whether or not he accepts your offer, but his alternative is to call your bluff and go to court.
As andy pointed out, a solicitor will give you a qualified opinon on this, but you could do worse than consult the CAB, which would cost you nothing.