Photo: RNZ / Quin Tauetau
Using AI algorithms to to renew expired Jobseeker benefits automatically could result in 'computer says no' discrimination, experts are warning.
Cabinet has agreed to introduce a suite of new obligations and sanctions for job seekers this year, and halving the time people can be on the benefit from a year to six months is among the proposed changes.
That would effectively double the workload for Ministry of Social Development staff processing re-applications and the ministry said it plans to use automatic decision making to roll these over.
MSD said such AI would not be used to decline Jobseeker Support benefits - that decision would be made by a staff member.
Victoria University of Wellington professor and AI expert Dr Simon McCallum said the potential for bias was too great.
"There are certainly areas to be concerned about," he said.
"Once an automatic decision is made the human won't think fresh about the situation, they'll just use whatever the computer says or default to a 'computer said no' situation."
McCallum said human empathy can be inhibited by such a process.
"When you are the human who just gets a group of people who are potentially problematic your mindset shifts into a 'I'm dealing with people who are problematic' and you get that expectation issue."
He said such automated decisions would need human oversight in a way that could make them less efficient.
AI expert Dr Karaitiana Taiuru specialises in the effect of automated decision making on indigenous and vulnerable communities.
He said it all comes down to the data and algorithms used and if that reflects the nuances of such groups of people.
"The data MSD has could very easily reflect bias against Maori, Pasifika and immigrants so in the ideal world companies and organisations would use those same communities to consult on the algorithms they're using, or how the automated system will work."
Taiuri said that should be the first step.
"MSD really needs to consult properly with all communities that will be impacted by this and I'm sure if they do this will be a cost saver but there has to be that human investment in the project first."
He also said a standard developed for MSD's use of such technology needed an overhaul.
"There is an issue here. MSD are saying they've got some safeguards from 2022 automated tools and AI have significantly changed in the last three years."
MSD said it is reviewing the standard in consultation with the privacy commissioner.
MSD is a co-signatory to The Algorithm Charter for Aotearoa New Zealand, which outlines the ways in which government agencies should demonstrate transparency and accountability in the use of data.
Under the charter, agencies agree to making sure their data is fit for purpose by understanding its limitations and identifying and managing bias - to name a few.
But Taiuru said the charter can not be enforced.
"AI in New Zealand is not being regulated so it is up to individual departments on how they handle automated AI systems and if they choose not to follow guidelines they don't have to."
MSD deputy chief executive for organisational assurance and communication Melissa Gill said it was important to make sure the ministry used automatic decision making carefully and responsibly.
"Our automated decision making process for regranting benefits uses a series of simple checks to check whether a client meets the legal requirements to have their benefit regranted."
She said the process did not include demographic information or profiling about people on benefits.
Gill said the automated process would not decline a benefit regrant, and beneficiaries would still be able to re-apply for their benefits in person.
"The redesigned 26-week reapplication and the use of automatic decision making in this way can make it easier for clients to do many of the basic things they need to do, without having to repeat information they have already given us, or repeat activities already completed. This allows us to focus on the interactions with clients that will help get them into work."
MSD said its Automated Decision Making Standard was introduced in 2022 after consultation with the Privacy Commissioner, the Human Rights Commission and the University of Otago Centre for Artificial Intelligence and Public Policy.
"It provides a framework for good practice and introduces specific automatic decision making safeguards, including the requirement that our clients get their correct entitlement and are not discriminated against," Gill said.
"Where appropriate, we are able to consult with external reference groups to help achieve this. MSD has access to a range of reference groups who we can involve when this be required."
Gill said MSD used the Standard to assess the shortened regrant process, and no specific concerns about bias or discrimination were identified.
"As a result, there was no need to consult with external reference groups."
AUT computer science professor and AI expert Dr Edmund Lai said some degree of bias is unavoidable in both human and computer systems.
He said using automatic decision making to rollover benefits at the expiry date should not be a problem.
"If the tool is used in exactly that way I don't see any problems because you're just saving manpower for screening all the routine cases where reapplication is successful and then a human will look at those other ones and rexamine those cases."
He said it would need human safeguards.
Law changes are needed in order for the government to introduce new obligations and sanctions for job seekers.
A new clause in the Social Services Amendment Bill, which has passed its first reading, vastly expands the decisions that can be made by automated systems to include sanctions.
MSD said it did not plan to use generative AI or automated decision making in that way.
Both the Salvation Army and Law Society have called for the clause to be scrapped, in submissions to the select committee considering the bill.
The Salvation Army's submission stated that automatic decision making "cannot account for the complexities we often see in the individuals we support", such as financial hardship, addictions, mental health issues or unstable living conditions.
The Law Society wants the clause allowing the expanded use of AI to be dropped entirely.
"This raises significant concern about how the use of automated systems will apply where the sanctions provisions involve some form of evaluative judgement, for example those relating to money management and community work."
The AI experts RNZ spoke to said while they supported using technology for good ways to automate decisions, using such AI for sanctions would come with risks.
Lai said it would not be an appropriate use of the technology to cut benefits or decide other monetary sanctions.
All agreed such use of the technology would need human oversight and that is not currently regulated.
Sign up for Ngā Pitopito Kōrero, a daily newsletter curated by our editors and delivered straight to your inbox every weekday.
