Using LLMs as a clinician support for detecting defense mechanisms from psychotherapy transcripts: a proof-of-concept study

This proof-of-concept study evaluated the feasibility of using large language models (LLMs) to support clinicians in identifying defense mechanisms in psychodynamic psychotherapy transcripts, comparing LLM assessments with those of psychodynamic therapists in both solo and supervision group settings. The first two sessions of a psychoanalytic case were analyzed. The transcribed sessions were segmented into excerpts and coded by two psychodynamic clinicians and two LLMs (ChatGPT and Gemini) guided with a structured prompt. Coding followed an adapted version of DMRS comprising 12 defenses organized into three maturity levels. For each excerpt, defense mechanisms were identified, and an overall defensive functioning index (ODF) was calculated. Inter-rater agreement was estimated using the Intraclass Correlation Coefficient (ICC), considering human-human, LLM-LLM, human-LLM, and integrated versions (clinical supervision vs. LLM integration) pairs. LLMs tended to assign a higher number of defenses, including more immature ones, producing profiles with lower average ODFs than clinicians, who instead provided more selective and, overall, more mature ratings. However, some of the defenses reported only by LLMs were subsequently recognized by supervised clinicians as clinically relevant and worthy of inclusion. Clinicians showed good to excellent inter-rater reliability, while ChatGPT and Gemini showed low inter-rater reliability, suggesting that the two models are not interchangeable. Agreement between clinicians and LLMs was low across the sessions. Therefore, the combined use of LLMs among clinicians is technically feasible and could potentially enhance understanding of patients’ defensive functioning, especially as a support for case formulation and supervision. Currently, LLMs cannot replace clinical judgment or the therapeutic relationship, but they might be considered as a support tool for clinical practice under human professional guidance.