Freedom of movement between Australia and the UK wasn't/isn't a 'right'...Freedom of movement for a certain age for a set amount of years. And we had the same right to go and work in Australia
You only had/have the 'right' to apply for a visa, which is subject to conditions and with no guarantee of approval...
Something that will soon apply to an additional 27 countries that we currently do have the right to exercise the 'right' of freedom of movement to and from!