Vivaan Echambadi

Financial Market Sentiment Analysis Using Large Language Models (LLM) and Retrieval-Augmented Generation (RAG)

Started Apr. 26, 2024

Abstract or project description

The stock market is a complex, non-linear, and time-variant system heavily influenced by public sentiment. Understanding the sentiment behind market movements can offer invaluable insights for investors, hedge funds, and financial analysts. Market sentiment analysis involves using machine learning techniques to interpret and quantify the emotions and opinions expressed in textual data, such as news articles and financial reports.

Recent advancements in machine learning and natural language processing (NLP) have enabled the use of large language models (LLMs) to perform sentiment analysis more effectively. Specifically, fine-tuned models like FinBERT are well-suited for financial sentiment analysis due to their ability to interpret domain-specific language. These models, combined with a Retrieval-Augmented Generation (RAG) pipeline, provide a robust framework to retrieve relevant financial news articles, process them for sentiment classification, and correlate sentiment scores with historical stock prices.

The proposed methodology includes extensive data preprocessing, such as vector embedding generation using Sentence-BERT, to ensure the quality of the dataset. Sentiment scores from FinBERT are used to predict next-day stock price movements, with findings showing that negative sentiment has a stronger immediate effect on prices compared to positive sentiment. However, the regression analysis reveals that sentiment alone explains only about 1% of the variance in stock price movements, indicating the influence of other factors such as market conditions, sector-specific trends, and macroeconomic indicators.

This paper aims to assess whether sentiment can serve as a reliable indicator for short-term stock price movements and how it can enhance trading strategies. The research underscores the value of integrating sentiment analysis with other financial indicators, offering a comprehensive approach to understanding market dynamics. For the purposes of this study, short-term stock price change is defined as the price movement from the market close on the day of the news publication to the market close on the following trading day. Future considerations include expanding data sources, incorporating real-time data processing, and addressing challenges such as model bias and data privacy to improve the robustness of sentiment analysis systems.