Perhaps they want to see what you are up to because they are paranoid that you are the kind of people who would smear dog poop
Perhaps if you mention to the chap "I couldn't help but notice your fence panels are a little warped, there are gaps showing. Need a hand knocking them back into place?" then he might take the hint.
I'm pretty sure you are getting more stressed about this camera business than your neighbour is. You need to go round, be friendly and politely talk to him about it. The worst that can happen is he can tell you to F-off. A lot of the neighbour grudges we have in this country could be solved if people just TALKED to their neighbours!
You might think he looks like a grumpy old fart, but he might really be a sweet old man who was under the delusion that you would appreciate footage of a burglar breaking into your house.