Deputy Prime Minister Winston Peters says company directors should have to declare whether they have been banned in other countries.
Winston Peters has started the week chairing Cabinet, as he begins to take over Jacinda Ardern's prime ministerial duties.
She was due to give birth to her first child on Sunday, and Mr Peters will become acting prime minister when she arrives at hospital for the birth.
Mr Peters told Morning Report the previous government did not take sufficient action after the Panama papers revealed the extent of foreign money in New Zealand-based trusts, and more vigilance was needed.
Yesterday, RNZ reported how British-born company director Matt Butterfield was banned from handling trusts in the tax haven of Guernsey for five years shortly before starting work as a trust and company director in Auckland.
Mr Peters told Morning Report he agreed that there should be an inquiry and a law change to make people declare if they have been banned in other jurisdictions.
"We just have not been careful enough.
"We've got to be far more vigilant, and far more untrusting of certain things that are being told to us as a government and as a country.
"These people are like rust, they never sleep."
He said yesterday Cabinet discussed a bill which would set out more scrutiny of people operating trusts.
"I'm pleased to say yesterday at Cabinet we had a bill setting out to prevent serious tax base erosion in this country.
"Law change is happening as we speak."
He said being acting prime minister was a privilege.
Ms Ardern is to take six-weeks maternity leave, but would continue to receive all Cabinet and Cabinet committee papers.
Mr Peters will run the day-to-day business of the government, and yesterday he chaired Cabinet and hosted Ms Ardern's weekly post-cabinet news conference for the first time.
On Shane Jones' attack on Fonterra
Mr Peters said it was totally legitimate to question the losses made by dairy giant Fonterra, given the amount of support the government gave the co-operative during the China botulism scare in 2013.
Regional Development Minister Shane Jones savaged dairy giant Fonterra last week, saying it was out of touch with the farming community and the the co-operative's should "catch the next cab out of town" and that it needed to be restructured.
Mr Peter's supported Mr Jones' comments at the time.
He told Morning Report Fonterra's Chinese venture, Beingmate, was a shocking idea and legitimate protests about it were not unreasonable.
He said its losses were as high as $1.383 billion and he and the taxpayers wanted some answers.
"We're all in this together, paying all the costs together because the dairy industry is one of the greatest thing this county has got going for itself if it's run properly."
On not giving support on three-strikes amendment
Mr Peters said he had a lot of respect for Justice Minister Andrew Little who he described as the "most reformist justice minister we have had for decades".
Justice Minister Andrew Little was forced to shelve a Cabinet paper repealing the controversial sentencing law, after what he said was a "rethink" by New Zealand First.
Mr Little has said NZ First was still committed to broader criminal justice reform - just not in "a piecemeal approach".
Mr Peters told Morning Report Mr Little was on the "right pathway, we just happened to have got something disconnected on the way through which is not the end of the world".
The three strikes law - passed in 2010 - set up a three-stage system of increasing consequences for repeat serious violent offenders.
On a third 'strike', the sentence must be the maximum possible unless the court considers it would be manifestly unjust.
It has been widely criticised by lawyers and academics.