I would say that you actually have 3 neighbours fences as in that corner is the house that has had all the work and extension done to it.
If you are building right up to the fence i might be an idea to think abut who has responsibility for the them for future replacement problems.
I would leave a gap for maintenance and its always then a good place for keeping bits stuffed out of sight.
I wouldn't tell them but would do whatever sound insulation you can to mitigate noise - which will be an added benefit of insulating for winter. And maybe dig that end lower before you start to remove the step-up then the neighbours will not see it.
If you are building right up to the fence i might be an idea to think abut who has responsibility for the them for future replacement problems.
I would leave a gap for maintenance and its always then a good place for keeping bits stuffed out of sight.
I wouldn't tell them but would do whatever sound insulation you can to mitigate noise - which will be an added benefit of insulating for winter. And maybe dig that end lower before you start to remove the step-up then the neighbours will not see it.