lmproved Diagnosis of Breast Cancer via NLP Analysis of Radiological Reports

Abstract

Toe main objective of this thesis was to evaluate the use of natural language processing (NLP) techniques and machine learning models to improve the specificity of breast cancer diagnosis and reduce false-positive rates using a dataset of radiological reports from Mexican hospitals. Toe methodology involved text preprocessing, feature extraction using NLP techniques and classification using machine learning models for the radiological reports. The preprocessing consisted of lemmatization, stop-word removal, and tokenization. Various NLP techniques were then applied, including bag-of-words, TF-IDF, Word2Vec embeddings, and Clinical­BERT embeddings. These were used as input features for classical machine learning models (Logistic Regression, Random Forest, Extreme Grading Boosting, Naive Bayes, k-Nearest Neighbors, Support Vector Machine and their ensemble) as well as a deep learning LSTM model. The models were trained, calibrated, and evaluated using metrics: AUC, accuracy, precision, recall, specificity and Fl-score. The key findings showed that the ensemble model with Bag-of-words and SVM using TF-IDF vectorized reports achieved the best performance, with an AUC of 0.79, specificity of 0.27 and AUC of 0.80 and specificity of 0.26, respec­tively. Thess model was able to identify all true positive cases while reducing the number of unnecessary biopsies by 19.49% and 15.08%, respectively. Feature importance analysis revealed that terms like "speculated", "irregular", and "4a category" were critica! for breast cancer classification. In contrast, the deep learning LSTM model performed poorly, with an AUC of only 0.52 and specificity of O. These results demonstrate the potential of NLP and machine learning techniques to enhance the reliability of breast cancer diagnosis and manage­ment, reducing the burden of unnecessary medica! procedures on patients and the healthcare system. The theoretical implications include the importance of effective feature engineering and the limitations of deep learning models for this specific task.