Explainable AI for trading 50 consumer discretionary stocks in the S&P 500

Abstract

This document presents a study that merges computer science techniques with finance, focusing on the development of an Explainable Supervised Machine Learning (SML) model aimed at achieving a balance between predictive accuracy and interpretability in price forecasting for Algorithmic Trading (AT). Utilizing SHAP (SHapley Additive exPlanations), both global explanations are provided to facilitate feature selection and determine the importance of various macroeconomic and technical indicators derived from historical data of 50 companies within the Consumer Discretionary sector of the S&P 500 Index. The study also employs hyperparameter tuning on lagged values to assess whether the price movements from one day can effectively predict subsequent market prices. Algorithmic Trading (AT) currently constitutes approximately 60% to 75% of total trading activity in U.S. equity markets, European financial markets, and major Asian capital markets (Groette, 2024). Projections indicate a significant growth trajectory for this sector. The driving force behind this expansion is the advancement of Artificial Intelligence (AI). As AI models incorporate more data, they tend to become increasingly intricate and opaque, evolving into what are commonly referred to as black box models. This complexity raises critical concerns surrounding explainability, interpretability, and transparency, as well as adherence to regulatory standards. Neglecting these issues can lead to severe market disruptions, including panic selling, liquidity evaporation, increased asset correlations, and a lack of clarity regarding the decision-making processes of AI models. Such challenges underscore the imperative for developing transparent and interpretable AI solutions in AT to mitigate risks and enhance market stability.