Yes and no. It's asbestos so it's very bad if you inhale it, but was usually only used to bind cementious (?) products, so should be safe unless exposed through damage. Still a lot less dangerous than blue asbestos though, which is the really evil killer stuff. Blue I believe was used as insulation.
Many threads on this already, have a search for the 'A' word.