Relax. For most Singaporeans, most if not all your CPF was amassed before you became a resident for tax purposes. In this case your CPF was contributed to on a pre tax basis prior to you being subject to Australian taxes and therefore not subject to Australian taxes as its not income, salary, capital gains, gain on sale, wages, tips or other taxable income. At the point where your CPF is remitted it becomes savings - and i can confirm the US IRS treats all CPF funds as such.
Its the Singaporeans who are tax residents in Australia who simultaneously earn in Singapore, contribute to CPF, and get paid interest have to answer to the tax authorities there because while resident in Australia for tax purposes, your worldwide income is subject to taxes. Those that do this are careful not to be tax residents for that fiscal year as only Australian sourced income is taxable in this case.
Assuming your first and last CPF contribution occurred before you became a resident for tax purposes, you should be fine. Only the interest paid while you were resident can only theoretically be applied in this case.