I'd press mortar all round the inside and outside where the ends of the duct are. If the gap is more than about a quarter inch, you can butter mortar onto the brick and push the duct through, if its very small, I just press it in with fingertips. As it is not a cavity wall there's no problem of excess mortar falling into the gap.
Spray the brick with water first.
Use a stiff brush to remove smears or excess from the face of the wall after the mortar has started to dry. After a few days hardening, you can remove the duct by pushing straight, the mortar doesn't stick to plastic much. If you rotate the duct while the mortar is fresh it will break the grip. This may enable you to pack mortar into any hollows or roughness. It is much easier to saw or trim the duct if you can slide it out of the wall to work on.
The cowl vent will cover the ragged edge of the hole unless it is very clumsily done. The duct should be sealed into the wall with mortar to keep draughts and insects out, the vent should be close to the wall but need not be sealed.
I believe harling is lime render with stone chips, I don't think we have it round here. Try not to take much off as I think it will be difficult to patch and look good. Silicone sealant should not be necessary.