If you knew any professional musicians you would know that a conductor does a lot more than just stand at the front waving his baton. That's just the visible culmination of many hours of work rehearsing the piece and helping the musicians to understand the way it has to be played. Orchestras can play without a conductor but they will just plod along without that personal interpretation that makes two performances of the same piece so different if you hear them both. Orchestras can like or dislike a conductor, if they dislike him they can make life hell for him in many subtle ways.