PhD Seminar by: Ryan Bluteau

 

    In this work we study how pruning weights can affect tabular neural networks based on the lottery ticket hypothesis. The lottery ticket hypothesis has been tested extensively in literature, often showing an ability to prune neural networks significantly. This work presents the same results for tabular neural networks using a wide range of tabular datasets. We show that we can significantly prune a model (some down to 1 neuron per layer) while improving accuracy for a large majority of datasets tested.

    However, this pruning approach presents challenges when applying it to larger pre-trained models (e.g. transformers). In particular, we lose the original weight initialization due to pre-training and training the original model from scratch or even finetuning is computationally expensive. Thus, we present work towards adapting our approach for these drawbacks. Using tabular neural networks, we present a second approach using a genetic algorithm which removes the need to train the original weights and focus on pruning the network. In addition, we apply a dataset reduction strategy to find lottery weights with as little as 5% of the original train dataset. Despite these reduction efforts, we still show an improvement over our original approach. Future work aims to apply this approach to transformers.

 

Internal Reader: Dr. Boubakeur Boufama

Internal Reader: Dr. Dan Wu

External Reader: Dr. Jonathan Wu

Advisor (s): Dr. Robin Gras