You are definitely being paranoid. I'm not saying sprinkle it on your cornflakes but the type of asbestos you are talking about is basically harmless. Even the HSE agree - they now treat it like any construction dust and fibre - and they're even more paranoid than you. I think you are mistaking it for other types of asbestos that are really nasty - pipe insulation and soft fibrous types. The stuff you are talking about is a different type of fibre and not the same at all.
What are the soft fibrous types? The only reason is I have heard that asbestos board has been used and found before acting as bath panels (search google) and you can See images etc.
How do you tell the difference between AIB and cement as both contain it right?