MSc Thesis Proposal "Adaptive Model Selection for Stock Market Prediction Using Apache Kafka and Microservice Architecture" By: Mohammad Ehsan Akhavanpour

Date: Thursday, September 14th, 2023

Time: 9:30 AM – 11:00 AM

Location: Essex Hall, Room #122

 

In today's globalized economy, financial markets are more interconnected than ever, generating vast

amounts of data from thousands of sources every second. The need to accurately analyze and

interpret this data is crucial for investors, analysts, and researchers alike. Traditional models for

market prediction are limited by their ability to adapt to the real-time nature and 'big data'

dimensions of these complex financial datasets. To address these challenges, this study introduces a

novel architecture that combines Apache Kafka with a microservices framework. This architecture

offers a scalable, real-time solution for financial market prediction that effectively manages the 5Vs of

big data: Volume, Velocity, Variety, Veracity, and Value. Apache Kafka's event-streaming capabilities

serve as the backbone of the architecture, enabling real-time data stream processing and distribution.

The system captures data from multiple sources in real-time and feeds it to various sinks, thereby

enhancing scalability and versatility. This real-time adaptation is optimized by an event-driven

approach, ensuring immediate updates across all layers of the architecture. One of the system's key

features is real-time model switching, which dynamically selects the most appropriate machine

learning model based on the market's current state, thereby maintaining prediction accuracy. Coupled

with Change Data Capture (CDC) mechanisms, this ensures that the data fed into the model is always

up-to-date. To enhance scalability while ensuring data quality, we employ a microservices architecture

in which each service operates independently and can be updated without affecting other services.

This provides high availability and fault tolerance, essential in a rapidly evolving financial environment.

By integrating Apache Kafka and microservices into a unified architecture that leverages real-time

event streaming and dynamic model switching, this study presents an innovative approach to tackle

the big data challenges in financial market prediction. The result is a system that not only

demonstrates increased scalability but also successfully maintains prediction accuracy through its realtime

model selection, making it an invaluable tool for financial market analysis.

 

Keywords: Apache Kafka, Microservices Architecture, Real-Time Model Switching, Financial Market Prediction

 

Internal Reader: Dr. Rueda

External Reader: Dr. Hassanzadeh

Advisor: Dr. Samet