Artificial intelligence can help gauge the effectiveness of cognitive behavioral therapy, according to new research from the University of Southern California published in PLOSOne.
Human raters typically assess the work of therapists-in-training using a list of 11 criteria. Engineering scientists at USC’s Signal Analysis and Interpretation Laboratory set out to explore whether AI could perform such work, utilizing transcripts from more than 1,100 real conversations with patients.
They found early success, with AI matching what a human evaluator could achieve at 73% accuracy, USC announced Nov. 10.
“Scaling up performance-based [cognitive behavioral therapy] evaluation to real-world usage opens up exciting opportunities towards the provision of fast and cost-effective feedback to the practitioner,” Nikolaos Flemotomos, a PhD student in electrical engineering at USC, and co-authors concluded. “Such feedback can be beneficial for training new therapists or for maintaining acquired CBT-related skills and can eventually lead to improved mental healthcare services and more positive outcomes for the patients.”
Flemotomos and colleagues believe this is the largest study to date focusing on automated evaluation of such therapy and first to use real people and conversations. One of AI’s biggest challenges in the process, those involved noted, was picking up multiple speakers and discerning meaning from text-based conversation. However, their creation did prove effective at assessing therapists’ interpersonal skills, how they structured sessions and their ability to establish rapport, among other factors.
USC experts believe their AI tool could be scaled to help bolster the CBT workforce and address growing demand for these services. Their goal is not to replace human supervision, however, only augment raters’ efficiency and help in self-assessment. Moving beyond text-based analysis to audible conversation is one avenue for further exploration.
“With improved speech technologies and employing a low-error transcription system, we expect the performance of [Cognitive Therapy Rating Scale] prediction systems to get better in the future,” the authors advised.
Marty Stempniak has covered healthcare since 2012, with his byline appearing in the American Hospital Association's member magazine, Modern Healthcare and McKnight's. Prior to that, he wrote about village government and local business for his hometown newspaper in Oak Park, Illinois. He won a Peter Lisagor and Gold EXCEL awards in 2017 for his coverage of the opioid epidemic.