The School of Computer Science is pleased to present…

Adaptive Model Selection in Stock Market Prediction: A Modular and Scalable Big Data Analytics Approach

 

MSc Thesis Defense by:

MohammadEhsan Akhavanpour

 

Date: Friday December 15, 2023

Time:  10:30 am – 11:30 am

Location: Essex Hall, Room 122

 

Abstract:

In today's globalized economy, financial markets are more interconnected than ever, generating vast amounts of data from thousands of sources every second. The need to accurately analyze and interpret this data is crucial for investors, analysts, and researchers alike. Traditional models for market prediction are limited by their ability to adapt to the real-time nature and 'big data' dimensions of these complex financial datasets. To address these challenges, this thesis proposes and implements a novel framework that combines Apache Kafka with a microservices framework. This framework offers a scalable, real-time solution for financial market prediction that effectively manages the 5Vs of big data: Volume, Velocity, Variety, Veracity, and Value. Apache Kafka's event-streaming capabilities serve as the backbone of the framework, enabling real-time data stream processing and distribution. The system captures data from multiple sources in real-time and feeds it to various sinks, thereby enhancing scalability and versatility. This real-time adaptation is optimized by an event-driven approach, ensuring immediate updates across all layers of the framework. One of the system's key features is real-time model switching, which dynamically selects the most appropriate machine learning model based on the market's current state, thereby maintaining prediction accuracy. Coupled with Change Data Capture (CDC) mechanisms, this ensures that the data fed into the model is always up to date. To enhance scalability while ensuring data quality, we employ a microservices architecture in which each service operates independently and can be updated without affecting other services. This provides high availability and fault tolerance, essential in a rapidly evolving financial environment. By integrating Apache Kafka and microservices into a unified framework that leverages real-time event streaming and dynamic model switching, this study presents an innovative approach to tackle the big data challenges in financial market prediction. The result is a system that not only demonstrates increased scalability but also successfully maintains prediction accuracy through its real-time model selection, making it an invaluable tool for financial market analysis.

 

Keywords:

Apache Kafka; Microservices Architecture; Real-Time Model Switching; Financial Market Prediction

 

Thesis Committee:

Internal Reader: Dr. Luis Rueda 

External Reader: Dr. Mohammad Hassanzadeh

Advisor: Dr. Saeed Samet

Chair: Dr. Muhammad Asaduzzaman