Date: Friday, October 23, 2020 

Time:  1:00 PM – 2:30pm 

Zoom url: https://zoom.us/j/95133871546? 

 

Recent advances in Natural Language Processing (NLP) field follows a trend of training gigantic models to achieve the state-of-the-art result, especially after the introduction of the Transformer architecture. ProphNet, PEGASUS, and GPT-3 are a few examples of these large models which either inherit whole or parts of the Transformer architecture. There is no doubt that these models perform well, but the cost of training and maintaining them is a real issue. Also, processing long sequences is still a challenge and will be achievable by adding even more parameters as the number of parameters grows quadratically with the size of the input. We are doing experiments on the sequence-to-sequence models to get similar performance with fewer parameters. We already obtained better results than the GPT-2 model with less than 90% of the original model’s total number of parameters in the text summarization task. Even though the results are not close to state-of-the-art, it shows that there is room for improvement. 

 

Keywords: Natural Language Processing, Text Summarization, Abstractive 

 

Internal Reader: Dr. Luis Rueda 

Internal Reader:  Dr. Dan Wu 

External Reader: Dr. Jonathan Wu 

External Examiner: TBD  

Advisor: Dr. Robin Gras