Better Naturalness Evaluation of TTS Systems

One of the main goals of Text-To-Speech systems is to generate natural speech. Therefore, a major evaluation criterion of TTS outputs is their naturalness, usually measured through a Mean Opinion Score (MOS). Naturalness is not a well-defined property, however. This paper decomposes naturalness into eight specific dimensions, based on how judges define the term. We then evaluate the outputs of the systems submitted to the Blizzard 2025 Challenge based on these new dimensions and compare the results with alternative evaluations of naturalness. This includes recent subjective human evaluations that were performed by the 2025 Blizzard Challenge organizers, as well as various automatic MOS methods. Based on this analysis, we propose to use five dimensions in place of a single, primitive notion of naturalness: Clarity, Fluency, Human-vs-Computer, Pronunciation, and Understandability. We propose that these would serve as the basis of a better evaluation framework for advanced TTS systems.