I was about to refer you to the mechanisms in the electoral commission for petitioning (the process of challenging) an election, but this was a non-binding referendum. I think (and I'm happy to be challenged) that the results of the referendum are now irrelevant, because the conservatives fought and won (however badly) a general election promising to deliver Brexit. So while criminals may be fined for cheating, the question of the Brexit referendum being open to challenge has been superseded by a general election.
To consider it another way.. David Camaron could have said - I'm ignoring the referendum, you are all stupid quitters, I'm calling a general election and my manifesto is to stay in the EU: a vote for us is a vote to ignore the referendum etc.. Had he won, then nobody could challenge his mandate. The same goes for Cable or Corbyn.
At the end of the day TM can do as she likes as long as she can get support in the commons to get it through.
So long answer is the EU cannot successfully sue us even if we voted remain and the govt of the day served Article 50 with sufficient member support. I might be wrong if there is an EU constitutional requirement for a referendum (as with some members) but I don't think we have one.