Hi there,
My bathroom extractor is very drafty.... so much so that during the winter I ended up not using it and taping over cardboard to stop the cold air getting in.
The reason for this, I believe, is that there is nothing behind the extractor, it's just mounted into a hole in the wall...