Well, if the roofer went ahead without giving you a price for the work (verbal or otherwise) then he has opened himself up to the possibility of not being paid the full amount he expected.
I don't see a problem with you not paying the invoice in full, after all it is his fault for not agreeing a price with you beforehand. He really should have done what I and most of the other traders in this country do - look at the job and supply the customer with a written quote and only go ahead with the work when there is an agreement on price.
FYI the trader doesn't need to provide a written quote if the work is urgent (ie leaking roof) and you instruct him to carry out the work and bill you afterwards (ie waive your rights to a written quote). But as you didn't instruct him to do the work, you don't have to pay the full bill if it's unreasonable.
It looks like you have saved a few quid and the trader has lost a wee bit.
No big deal here, we are talking pennies really.
On a side note, perhaps the OP could have a read of this page and perhaps explain why she feels £180 is excessive? -
http://quotationcheck.com/dissecting-a-quotation/
Maybe this should be a lesson for both of you? The homeowner should always be firm with the trader and make it perfectly clear that no work should carried out and only a free written quote is required at this time. The trader should not go ahead with any work until a quote has been delivered and accepted.
The trader has lost a bit of money but the homeowner has had to deal with the stress of seeking legal advice an spending time posting questions on help forums etc so both have lost out really.