TrustAI: An Expert-Based Methodology for Evaluating and Certifying the Trustworthiness of Artificial Intelligence Systems

Despite the acknowledged importance of ensuring the trustworthiness of AI systems, there is currently a lack of well-established methods and techniques for evaluating and certifying their trust. To address this issue, we propose TrustAI, a novel expert-based methodology for evaluating and certifying the trustworthiness of AI systems. Our approach is based on a set of high-level requirements, i.e., the pillars of trust, which are decomposed into a set of low-level criteria expressed in the form of questions that are assessed by AI experts. It aggregates the expert judgments through a hierarchical model in order to provide a quantitative expression of the level of trust (LoT) of a given AI system, and generates a certificate in case that the evaluation results are satisfactory. Fuzzy logic is utilized for modeling the uncertainty of expert judgments and for defining the parameters of the model. The proposed methodology is highly configurable, enabling the generation of custom evaluation models tailored to specific application domains and regulatory frameworks. It is illustrated through a use case on two actual AI systems, evaluated using a model derived via the methodology based on a recently issued regulatory framework.