I'm grateful to my teammates Blake Hu, Julian Baldwin, Stephen Cheng, Marko Veljanovski, and Michelle Zhang for their dedication, creativity, and collaborative spirit, and to our advisor Prof. Zach Wood-Doughty, whose feedback shaped both the research direction and the way we presented it.
This one taught me how to balance a theoretically interesting idea against practical efficiency constraints — and put me through a bit of light CUDA hazing along the way.
Abstract
Fine-tuning is the standard way to adapt a pre-trained language model to a downstream task, but the time and energy needed to update every parameter become prohibitive as models grow. Recent parameter-efficient methods shrink the number of parameters that get updated without meaningfully shrinking training time or energy consumption.
We propose a time-efficient alternative based on feature extraction: treat an off-the-shelf language model as a fixed source of embeddings and train a small feed-forward network on top of it for each downstream task. Averaged across the GLUE NLI benchmark with distilRoBERTa, this trains 124× faster than full fine-tuning and 101× faster than parameter-efficient fine-tuning, while reaching 81.9% and 85.0% of their respective performance.