I wouldn't worry about breaking any more of that concrete away, if you've a solid section of pipe there you've already exposed, cut through it there, (leaving enough to get a coupling onto!), then just break the rest of the old stuff and pot out, (making sure you dont damage the section you need to leave in situ!) Once you've got all that out, offer the new gulley in, and connect up. I would suggest some lean mix concrete under and around the base of the gulley to support and hold it in position, then bed pipe in peagravel, before backfilling. Try and avoid big lumps or stones in the initial backfill if possible.
Benching in the chamber also looks in poor condition, ideally wants redoing, although as a shared sewer, it is the Water Company's responsibility, may be worth asking them first if they wish to look at it/carry out remedial works.