Photo: RNZ
A mega database critical to the government's social investment approach has required a $2 million upgrade because it is creaking at the seams.
The Integrated Data Infrastructure, run by Statistics New Zealand, is a large anonymised database holding "microdata about people and households" sourced from government agencies, surveys and NGOs.
It is often used by government agencies and academics for research purposes under strict conditions.
Ministerial documents released to RNZ under the Official Information Act show that while the database is considered the "primary tool" to support the government's social investment analysis, it is "outdated", receiving little investment since it was first developed in 2011.
"In its current state it's not well placed to support quick turnaround analysis or a scaling up of its use - the existing infrastructure is dated and insufficiently resourced," Social Investment Minister Nicola Willis was warned in a briefing.
There were also "notable gaps" in the IDI when it came to Māori, Pacific people, children and disabled people.
An upgrade should be an immediate priority "to ensure its ongoing viability", the Social Wellbeing Agency (now Social Investment Agency) briefing said.
The advice was heeded and in October, the government drew $2 million from the $50.5m allocated in Budget 2024 for social investment to carry out an upgrade of the IDI.
This included adding more data, improving server capacity to reduce processing demand and enable faster analysis, improving access, and working to "determine how we can promote a shift to data-driven decision-making across government", Statistics Minister Andrew Bayly said in a statement.
"Unfortunately, like a lot of infrastructure, the IDI suffered from underinvestment under the last government and while it is fully functional and available to government agencies, academics and NGOs to use, Stats [NZ] has identified that investment and improvements are needed to improve performance."
The IDI was "central" to the government's social investment ambitions but the last time it had received a significant investment was in 2016, when it got $265,000, Bayly said.
"Stats has done a great job of resourcefully maintaining the IDI since then, but we are realistic about the fact there is growing need for improvement.
"Work is already underway to understand and plan for this work and, while I am still considering advice I won't comment in detail, except to say that the IDI will be resourced to meet future requirements."
The upgrade was welcomed by Auckland University social sciences associate professor Barry Milne, who regularly uses the database for his research.
He described the IDI as "a bit creaky at times" and said there were occasionally outages where it was not working at all.
While the use of data-driven analysis was important, he cautioned against the IDI being the main driver of the government's social investment approach.
"It is really just focused on interactions with the government, so you'll know if people are using fewer services, which will obviously save the government money, but you don't know if their wellbeing has necessarily improved."
Identifying need and working out how to best assist that need were two crucial elements to a good social investment programme, Milne said.
"The easy bit is identifying people who will have later problems. It's actually finding programmes that are doing to help them, and implementing those programmes without the political whim of them being taken away at some later point."
The government has acknowledged the risks of solely relying on data, with Willis noting in a Cabinet paper in May that measuring other outcomes is also important.
Sign up for Ngā Pitopito Kōrero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.
