Skip to content

Latest commit

 

History

History
45 lines (35 loc) · 1.78 KB

README.md

File metadata and controls

45 lines (35 loc) · 1.78 KB

Natural language processing course 2023/24: Parameter-Efficient Fine-Tuning of Language Models

Objective of Project

This project is conducted within the course "NLP" at University of Ljubljana (FRI). The objective is to implement and compare different PEFT (Parameter-Efficient Fine-Tuning) Methods for differenft NLP tasks. In our analysis, we will compare the following three methods:

  1. LoRA
  2. Soft Prompting
  3. IA3
  4. BitFit.

Overview of the Tasks

We will compare the methods based on the following benchmarks and their respective tasks:

Benchmark NLP Task
CommonsenseQA Commonsense Reasoning
CoNLL-2012 Coreference Resolution
XSum Text Summarization
SST5 Sentiment Analysis
Slovene SuperGLUE Slovene BoolQ (Boolean Questions)

Project Structure

The project structure is organized as follows:

  • data/: Contains the data.
  • report/: Contains the latex report of our results.
  • src/dataset_handler: These dataset handlers are used for reading and processing the data according to their objectives.
  • src/evaluator: Trains and evaluates the different models.
  • src/trainers: Every task has its own trainer specified in this folder.
  • src/utils: Contains helper functions.
  • main.py: Main script that runs the training and evaluation process.
  • output/: Contains the output (trained models and metrics).

Running

cd src
sbatch run.sh

Authors

  • Ondra Komin
  • Andrej Susnik
  • Eileen Vu