Beyond Surface Similarity: Evaluating Structured Output Consistency in Large Language Models

As Large Language Models (LLMs) become central to production systems requiring structured data generation, evaluating output consistency remains an open challenge. Existing metrics—designed for natural language text—fail to capture the semantics of structured formats like JSON, leading to both false positives and false negatives. We present an evaluation framework with two contributions: (1) a semanticstructural similarity metric that recognizes equivalent representations while detecting breaking changes, and (2) a consistency scoring methodology that aggregates pairwise measurements to quantify model reliability. Through experiments on 2,400 synthetic test cases with controlled variations, we show that our metric scores 0.86–0.90 for semantic equivalents while scoring near-zero for structural violations (flattening: 0.051, nesting: 0.000)—whereas BERTScore (0.90–0.99) and DeepDiff (0.78–0.81) assign high similarity to both, failing to distinguish them. Benchmarking six LLMs reveals that Claude-3.7-Sonnet maintains near-perfect structural reliability even at T = 0.9, while other models degrade substantially. The framework enables informed decisions about model selection, temperature configuration, and prompt design for structured output tasks.