You’re slightly off in your quote. It should be ‘Rugby is a game for hooligans played by gentlemen. Football is a game for gentlemen played by hooligans. Cricket is a game for gentlemen played by gentlemen’.
I watch a lot of rugby and regularly attend professional and amateur club games. I’ve not heard any skin colour related abuse that can remember, but there is lots of nationality related banter in international matches that can get close to the line but tends to be self policed by the crowd If it starts going too far.