Automated radiology report generation using radiomics and natural language processing techniques

Abstract

This thesis addresses the significant challenges in breast cancer diagnosis in developing countries, where delayed follow-ups due to resource constraints can impede timely and accurate detection, affecting patient outcomes. A novel approach using radiomic features integrated with transformer models to automate mammography report generation, specifically focusing on report conclusions is proposed. The primary goal is to assess if these AI-driven models can replicate the diagnostic accuracy of expert radiologists in assigning BI-RADS categories and recommending follow-ups or biopsies. The study begins with meticulous image preprocessing, including a customized histogram matching scheme to standardize input data and reduce variability among images from different vendors. Radiomic features were then extracted and validated through a classification task obtaining an AUC of 0.81, proving their efficacy as inputs for the transformer architecture. The transformer models utilized both radiomic features and deep learning features extracted via a pretrained CNN. This approach allowed for a direct comparison of model performance between the hand-crafted radiomic inputs and the more complex deep learning features against expert evaluations. Results showed that the models reached high agreement with radiologists’ evaluations, with kappa values reaching up to 0.93 for the simpler BI-RADS categorization task (1 & 5) using deep learning features. However, performance declined in more complex cases, with kappa values dropping to 0.23 for radiomic features across all BI-RADS categories (1, 2, 3, 4 & 5), indicating only fair agreement. In contrast, deep learning features maintained a moderate agreement with a kappa of 0.41. Despite these promising results, the study acknowledges certain limitations, including the inability to fine-tune feature extraction due to the hand-crafted nature of radiomic features, as well as the potential subjectivity in the data, given that radiologist evaluations are susceptible to human error. Nonetheless, this research lays crucial groundwork for future AI advancements in radiological diagnostics, aiming to enhance the efficiency, accuracy, and comprehensiveness of medical image analysis in resource-limited settings.