Sure, object on the grounds that it looks awful, which standard for aesthetics will you be applying your objection under?
Party wall, no, brick fence.
I'm simply being pragmatic, unlike you, if he wants to keep it nothing will ultimately stop him.
People can do all kinds of things you might object to based upon 'it is ugly' which is really another way of saying 'I demand that others do what I deem acceptable on their land', again, good luck doing anything productive with that attitude.
Ultimately you could spend a great deal of time and effort getting him to remove it, for him to simply re-build it 1cm away from the wall, which achieves what exactly?
It is not clear to me the status so the OP might do well to contact the Land Registry and Citizens Advice for clarification. Apparently it is NOT a party wall, it is on the boundary for which the OP is responsible. Therefore he has no right to build anything there. The OP asked him to build a wall, providing the bricks. Does the OP have a written agreement, or receipt? If not, after 10 or 12 years (I forget) the neighbour could end up claiming that he owns the wall and it is on his land. Unless the OP can prove that he paid for the wall to be built. As to the chimney, that is illegal, but getting it removed might be expensive. The courts are awful when it comes to boundary disputes, really it is a job employment scheme for greedy and conniving lawyers. I know, I've dealt with some of these leeches. Some are honest, but many act in an immoral manner, they know how to play alond and pocket the dosh. The law is unclear and expensive to apply.