Yeah, it sounds like the grout isn’t holding up well in the wet areas, which could be a mix of things. Since you mentioned skipping some of the slaking time, that might’ve kept the grout from fully hydrating before application, which can make it weaker. Another possibility is too much water...