Optus has paid a six-figure fine after a technical glitch let scammers gain access to customers' bank accounts.