MSc Thesis Proposal "Adaptive Model Selection for Stock Market Prediction Using Apache Kafka and Microservice Architecture" By: Mohammad Ehsan Akhavanpour

Thursday, September 14, 2023 - 09:30

The School of Computer Science is pleased to present…

Thesis Proposal Announcement

Adaptive Model Selection for Stock Market Prediction Using Apache Kafka and
Microservice Architecture
MSc Thesis Proposal by: Mohammad Ehsan Akhavanpour
Date: Thursday, September 14th, 2023
Time: 9:30 AM – 11:00 AM
Location: Essex Hall, Room #122
In today's globalized economy, financial markets are more interconnected than ever, generating vast
amounts of data from thousands of sources every second. The need to accurately analyze and
interpret this data is crucial for investors, analysts, and researchers alike. Traditional models for
market prediction are limited by their ability to adapt to the real-time nature and 'big data'
dimensions of these complex financial datasets. To address these challenges, this study introduces a
novel architecture that combines Apache Kafka with a microservices framework. This architecture
offers a scalable, real-time solution for financial market prediction that effectively manages the 5Vs of
big data: Volume, Velocity, Variety, Veracity, and Value. Apache Kafka's event-streaming capabilities
serve as the backbone of the architecture, enabling real-time data stream processing and distribution.
The system captures data from multiple sources in real-time and feeds it to various sinks, thereby
enhancing scalability and versatility. This real-time adaptation is optimized by an event-driven
approach, ensuring immediate updates across all layers of the architecture. One of the system's key
features is real-time model switching, which dynamically selects the most appropriate machine
learning model based on the market's current state, thereby maintaining prediction accuracy. Coupled
with Change Data Capture (CDC) mechanisms, this ensures that the data fed into the model is always
up-to-date. To enhance scalability while ensuring data quality, we employ a microservices architecture
in which each service operates independently and can be updated without affecting other services.
This provides high availability and fault tolerance, essential in a rapidly evolving financial environment.
By integrating Apache Kafka and microservices into a unified architecture that leverages real-time
event streaming and dynamic model switching, this study presents an innovative approach to tackle
the big data challenges in financial market prediction. The result is a system that not only
demonstrates increased scalability but also successfully maintains prediction accuracy through its realtime
model selection, making it an invaluable tool for financial market analysis.
Keywords: Apache Kafka, Microservices Architecture, Real-Time Model Switching, Financial Market Prediction

Thesis Committee:

Internal Reader: Dr. Rueda
External Reader: Dr. Hassanzadeh
Advisor: Dr. Samet