Usability testing, designed to improve products such as cell phones and Wii remotes, could also work for human services like foster care, researchers say.
The way companies use a small number of consumers to test merchandise prior to mass production could not only diagnose what works and doesn’t work with a new intervention for families, but it could also encourage problem-solving and collaboration.
Business leaders have long understood the value of subjecting a product to intended and unintended uses before moving to full-scale production.
“That wrist strap on your video game remote is probably the result of an exuberant child accidentally flinging a prototype through a TV during pilot testing,” says Karen Blase, a senior scientist at the Frank Porter Graham Child Development Institute at the University of North Carolina at Chapel Hill. “A strap is a lot easier to add before you ship three or four million remotes.”
Despite usability testing’s history in the corporate sector, within the fields of program evaluation and applied social science, the testing is new and under-researched.
Blase and colleagues believe that before widely putting an intervention in place, usability testing could create a critical window that would allow for analysis and improvement.
Their findings are published in the journal Evaluation and Program Planning.
Usability testing put to the test
In 2010, the federal government created the Permanency Innovations Initiative (PII) to improve outcomes among children in foster care and designated six grantees with innovative interventions designed to help children find permanent homes.
To determine usability testing’s effectiveness in a human services setting, the researchers examined a new intervention from one of these grantees: The Kansas Intensive Permanency Project (KIPP), a public-private partnership of the University of Kansas, the state’s Department for Children and Families, and the state’s network of private providers of foster care.
KIPP was establishing a new intervention for families in order to help children with “serious emotional disturbance” find permanent homes within three years. These children are more than three times as likely to remain in long-term foster care as their peers.
The new intervention aimed to enhance effective parenting practices to reduce the need for foster care. Usability testing assessed the viability and acceptance of the new intervention as the first families began to participate.
Just as the addition of a simple wrist strap has saved untold televisions from hurtling remotes, the researchers found that usability testing identified several challenges for KIPP leaders to address in order to avoid later unwanted consequences.
First, the testing identified the types of families least likely to accept the new intervention. Administrators then devised strategies that brought the participation rate up to their 70 percent target.
A seven-workday timeframe for initial “evaluation assessments” was not feasible due to the need to schedule multiple parties while addressing logistical obstacles, usability testing showed.
As a result, KIPP administrators adjusted their expectations, doubled the timeframe, and relieved families and professionals of an unnecessarily stressful deadline.
Usability testing told KIPP what was working. Because the intervention required video recording each session, administrators were concerned that families might not engage in the process.
However, nearly every parent that agreed to participate in the new intervention continued through several sessions. Although usability testing revealed that parents with older youth were less likely to consent to the intervention, overall, on measures of engagement with families the intervention exceeded expectations.
Front-line staff are encouraged to report to leadership about procedures that didn’t work. In turn, staff combined efforts with their immediate supervisors and administrators to fix early problems. With such collaboration, procedural changes improved engagement with families, including new talking points tailored for parents based upon the age of their child.
The researchers see potential for more applications for usability testing in human service settings.
“Making small tests of change, which translate to improvements early on, can prevent us from walking too far down a path that won’t lead us out of the woods.”
Becci A. Akin from the University of Kansas was a co-principal investigator on the study.
Source: UNC-Chapel Hill