A Multimodal Learner-State Recognition Module for Adaptive E-Tutoring
MSc Thesis Proposal by: Shobhit Kumar Biswal
Date: Wednesday October 28, 2026
Time: 2PM
Location: Essex Hall 122
Abstract:
Virtual e-tutoring systems often rely on predefined dialogue rules or text-based assessments that provide limited insight into a learner’s understanding and emotional state. This research proposes a multimodal learner-state recognition module that combines speech-to-text transcription, spoken emotion recognition, and answer evaluation to support adaptive pedagogical feedback. The module processes a learner’s spoken response, assesses answer correctness and confidence, and identifies emotional cues from speech. These signals are combined through a pedagogical decision-making mechanism to select an appropriate tutoring response, such as corrective feedback, encouragement, a hint, or a worked explanation. The research focuses on the design, implementation, and evaluation of this modular approach, including the accuracy of its individual components and the consistency of its tutoring decisions. The intended contribution is a framework for more context-aware and responsive interaction in virtual learning environments.
Keywords: Multimodal Learning; Speech Emotion Recognition; Learner-State Recognition; Adaptive E-Tutoring; Pedagogical Decision-Making
Thesis Committee:
Internal Reader: Dr. Zouchen Zhang
External Reader: Dr. Olena Syrotkina
Advisor: Dr. Xiaobu Yuan
