MelAnalyze: fact-checking melatonin claims using large language models and natural language inference

Abstract

With the explosion of health related information in mainstream discourse, distinguishing accurate health-related claims from misinformation is important. Using computational tools and algorithms to help is key. Our focus in this paper is on the hormone Melatonin which is claimed to have broad health benefits and largely sold as a supplement. This paper introduces ’MelAnalyze,’ a framework for using generative and transformer-based deep learning models adapted as a natural language inference (NLI) task, to semi-automate the fact-checking of general melatonin claims. MelAnalyze is built upon a comprehensive collection of melatonin-related scientific abstracts from PubMed for validation. The framework incorporates components for precise extraction of information from scientific literature, semantic similarity and NLI. At its core, MelAnalyze leverages pretrained NLI models that are fine-tuned on melatonin-specific claims along with semantic search based on vectorized representation of the articles. The models are tested on melatonin claims from Google and Amazon product description to evaluate the system’s utility.The best model, fine-tuned versions of LLaMA1, attain good precision, recall, and F1-score of 0.92. We also introduce a web-based prototype tool to visualize evidence and support fact-checking algorithm evaluation. In summary, we show MelAnalyze’s role in empowering users and researchers to assess melatonin-related claims using evidence-based decision-making.