You can soon tell which of the above two it is. Press and release your brake a few times (with the engine running) whilst stationary. Does it still hiss? IF so, it's likely to be a servo problem. The OP says it happens when the pedal is "RELEASED" though. That could be normal - just some sound deadening missing or something. Try the following.
Switch the engine off. Whilst stationary, press and release the footbrake a dozen or so times. The feel of the pedal should change. It will go harder. Then rest your foot gently on the brake pedal and start the engine. As the engine catches, you should feel the pedal get "pulled" down away from your foot a bit. That suggests the servo is working normally. Finally, with the engine running, press the brake. Does the engine falter or the engine note change appreciably?