The School of Computer Science is pleased to present…
An Approach for Lip Synchronization in ECA for E-learning
MSc Thesis Proposal by: Rajveer Singh
Date: Tuesday, February 27th, 2024
Time: 3:30 PM – 5:00 PM
Location: Essex Hall Room 122
Abstract:
Embodied Conversational Agents (ECAs) are becoming more and more incorporated into diverse sectors such as customer service, healthcare, education, and entertainment. They are utilized to improve user experiences and make interactions more efficient. Lip synchronization and Human-Like expressions play a crucial role in enhancing user trust, satisfaction, and overall anthropomorphism of ECAs. By ensuring that the lips of an ECA correspond with its spoken words, lip synchronization improves the naturalness and perceived realism of interactions. Through expression manipulation, ECAs may use their facial expressions to communicate intents and feelings, leading to more empathic and engaging conversations. When combined, these characteristics improve user engagement, understanding, and rapport in human-computer interactions by making ECAs more believable and successful overall. Their smooth integration improves the ECA's capacity to forge deep relationships and accomplish its communication objectives.
This study presents a novel approach to achieving accurate lip synchronization and facial expression manipulation utilizing machine learning and 3D animation. The algorithm uses a pre-trained multilingual machine learning model for real-time phoneme representations for Text-to-Speech generation and lip synchronization. 3D object files are used for illustrating phonemes derived from the machine learning model. These files are utilized to interpolate between different lip forms and to monitor and map the motions of the lips. This process ensures precise synchronization of the avatar's lip movements with the produced speech. Additionally, the algorithm incorporates ECA’s facial expression manipulation enabling the ECA to express happiness, surprise, concern, etc. increasing audience engagement. Through the integration of the newly developed algorithm with the overall E-tutoring system architecture, this research contributes to the advancement of interactive online learning experiences.
Thesis Committee:
Internal Reader: Dr. Olena Syrotkina
External Reader: Dr. Leo Oriet
Advisor: Dr. Xiaobu Yuan