PhD Thesis Proposal - Advanced Task-Specific Transformer Pruning by Ryan Bluteau

 

This proposal aims to advance text classification in transformers, specifically tested on sentiment classification. Current trends, e.g. large language models (LLM), are increasing transformer and dataset sizes to obtain better accuracy. This trend has led to a dependency on large GPU clusters and higher costs to operate/maintain (compounded by prediction speed), often locking advancements and models to companies embedding innovation while controlling the future of AI.

      This proposal focuses on building a path to smaller models by applying pruning strategies to transformers that is applied to text classification tasks (sentiment classification). The proposed approach first focuses on data acquisition. The text data is collected from large datasets with millions of samples (however it is not limited to this approach). Then the data is augmented through translation and text generation. All data is labelled using auxiliary labels (emotions in this case) from a 0-shot model, which is compressed into a stronger label (sentiment in this case).

      The second stage is model pruning (we use BERT) with the goal of maintain accuracy in the classification task. Typically, BERT has been pruned to 40-50% its original size with little impact on accuracy (whether distilled or pruned). The proposed approach aims to exceed this using a genetic algorithm to improve how we apply the lottery ticket hypothesis, while using our lottery sample hypothesis that we developed. The aim is to integrate our collected and 0-shot generated datasets to the approach to make a general path for any task (particularly lacking any labeled data).

 

Internal Reader: Dr. Boubakeur Boufama

Internal Reader: Dr. Dan Wu       

External Reader: Dr. Jonathan Wu

Advisor(s): Dr. Robin Gras