Sounds like a lesson in learning not to build right up to your boundary. You're at their mercy.
You'd need some patient use of a trowel to get that lot out now, after removing their fence.
But... it will fill up with something - dead leaves, dead birds, moss, whatever. If it's all or mostly gravel then it may be beneficial.
I'm guessing you would have unfinished pointing facing them, against the old wall where it was inaccessible when built, so they felt the need to cover it up. I had a neighbour built a garden wall before I moved in, just left dribbling slops hanging out on my side. If I'd been your neighbour and you'd talked to me I'd have said just get rid of the old wall and point it down to the ground, then have nothing there. But it's more complicated as there's a housing association involved.
I don't think there's a good solution, perhaps what they've done is the best anyone could do given that you built an exposed empty cavity that would have been ugly on their side if they'd removed the old wall.