err I may have forgotten my law lectures but what are you all on about...was there a restriction in the original planning permission for the pool to specify opening hours, its use as a commercial operation etc etc? if not this is just a calm the morons party piece.
See all the endless twaddle actions...the pub next door got planning permission for a kitchen extension...now they have more clients and i don't like it ..well tough.... The restaurant got planning permission for tables outside...now people sit outside and eat a meal...I don't like it well tough.
Unless there was a restriction placed on the original permission eg planning or license as to the volume, hours or days of use then any business or public body or individual can carry on with its business regardless. If you vary the activity of the business..eg a pub that now wants to serve food then the there needs to be a variation of use.
In your case swimming before roof , swimming after roof...end of.