Incorporating evidence into mental health Q&A: a novel method to use generative language models for validated clinical content extraction Kharitonova, Ksenia Pérez Fernández, David Gutiérrez Hernando, Javier Gutiérrez Fandiño, Asier Callejas Carrión, Zoraida Griol Barres, David Question-answering Generative language model Large language model Generative language models have changed the way we interact with computers using natural language. With the release of increasingly advanced GPT models, systems are able to correctly respond to questions in various domains. However, they still have important limitations, such as hallucinations, lack of substance in answers, inability to justify responses, or showing high confidence with fabricated content. In digital mental health, every decision must be traceable and based on scientific evidence and these shortcomings are hindering the integration of LLMs into clinical practice. In this paper, we provide a novel automated method to develop evidencebased question answering systems. Powerful state-of-the-art generalist language models are used and forced to employ only contents in validated clinical guidelines, tracking the source of the evidence for each generated response. This way, the system is able to protect users from hallucinatory responses. As a proof of concept, we present the results obtained building question-answering systems circumscribed to the clinical practice guidelines of the Spanish National Health System about the management of depression and attention deficit hyperactivity disorder. The coherence, veracity, and evidence supporting the responses have been evaluated by human experts obtaining high reliability, clarity, completeness, and traceability of evidence results. 2024-05-14T07:54:23Z 2024-05-14T07:54:23Z 2024-03-03 journal article Ksenia Kharitonova, David Pérez-Fernández, Javier Gutiérrez-Hernando, Asier Gutiérrez-Fandiño, Zoraida Callejas & David Griol (03 Mar 2024): Incorporating evidence into mental health Q&A: a novel method to use generative language models for validated clinical content extraction, Behaviour & Information Technology, DOI: 10.1080/0144929X.2024.2321959 https://hdl.handle.net/10481/91737 10.1080/0144929X.2024.2321959 eng info:eu-repo/grantAgreement/EC/NextGenerationEU/TED2021- 132470B-I00 info:eu-repo/grantAgreement/EC/H2020/823907 http://creativecommons.org/licenses/by/4.0/ open access Atribución 4.0 Internacional Taylor & Francis Group