Leveraging Local-LLM for sentiment analysis to enhance text data quality assessment in information science research

This study explores how a privacy-preserving local large language model (Local-LLM) assists researchers in evaluating unstructured text data quality, early in the analysis process. It aims to accurately and effectively determine if a Local-LLM can evaluate the quality of open-ended survey responses compared to human coders during the initial analysis of unstructured survey data. A Python tool using Llama 3.2 and 3.3 classified 604 survey responses. Human-coded sentiment labels served as a baseline; model performance was assessed with confusion matrices, F1, Cohen’s κ and Gwet’s AC1. All processing was offline to protect data privacy. Llama 3.3 achieved top performance (F1 ≈ 0.97, AC1 ≈ 0.97), while Llama 3.2 also excelled on consumer hardware. Automated sentiment analysis reduced processing time from 4 h to 8 min and identified short responses that manual reviewers might miss, improving speed and data quality. The approach’s reliability beyond English or on longer narratives remains to be examined in future work. The Local-LLMs allow researchers to rapidly filter and assess large volumes of unstructured text data before conducting deeper analysis. This study demonstrates that Local-LLM’s semantic features can be used for sentiment analysis as a scalable, cost-effective and rapid method for evaluating unstructured text data, aligning with the research questions before analysis. It also illustrates how to use zero-shot prompting to interact with Ollama via Local-LLMs, simplifying AI integration for non-technical researchers.