A method for evaluating diagnostic effectiveness using algorithms based on opinion obtained from artificial intelligence models

Objective: To develop a method for assessing the effectiveness of diagnostics using differentiated algorithms based on opinion, obtained from AI models, compared to the effectiveness of conventional diagnostics. Material and methods. Two routing scenarios for diagnosing malignant and benign skin neoplasms (MNs and BNs) were simulated. The first scenario involves diagnostics using algorithms based on the conclusions of the Derma Onko Check AI model. The second scenario involves standard diagnostic routing by general practitioners, internists, and dermatovenerologists. The diagnostic effectiveness indicators used in the simulation, including those of general practitioners, internists, and dermatovenerologists, were obtained from previous clinical studies. The modeling was conducted using clinical data and photographic images from 90 patients with skin cancer (39 melanomas and 51 basal cell carcinomas) and 291 patients with skin cancer (100 non-melanocytic skin tumors and 191 melanocytic skin tumors). Results. To assess the relative diagnostic efficiency using algorithms based on AI model output, calculation formulas with visualization of the results in the form of a quadrant matrix were developed and proposed. An mobile app called “AI-diagnostic efficiency calculator” (CalcRDAI&RNDAI) was developed for practical use to automatically calculate diagnostic efficiency indicators using algorithms based on AI model output. When testing a method for evaluating the use of algorithms based on the findings of the Derma Onko Check software, a 1.9-fold increase in the detection of skin cancer cases and a 10.5-fold decrease in missed cases were observed. The evaluation results are located in quadrant I (more detected, fewer missed), confirming the value of the diagnostic algorithm using algorithms based on the findings of the Derma Onko Check AI model in providing medical care. Conclusion. The proposed method for evaluating diagnostic effectiveness using the developed formulas for calculating effectiveness and visualizing the results in a quadrant matrix enables objective evaluation of the effectiveness of AI models in multi-stage diagnostic routing.