Damp can be caused by a myriad of problems and you really need to view the rooms in question both internally and externally to make a sound judgement but it sounds like a ventilation problem [internally ].If you have double glazed windows do they have vents in the opening lights? also do you have wall/cavity ventilation?does your internal plastering go down to the floor? [externally ] is your pointing in good condition? is your DPC in good condition and continuos?is water ponding/holding near your brickwork?A temporary solution is to wash infected walls down with an antifungicidal wash and then prime with an antifungicidal primer this would only address the sympton and not the problem i hope this is helpful