This repository contains code for Model Training and Evaluation for the BOLI project. I train and evaluate 3 types of models, namely:

1. Classic ML Algorithms (SVM, Random Forest, XGBoost)
2. BERT-based model MuRIL fine-tuning
3. MuRIL with Adapters fine-tuning

training
|_
|_
|_ muril-fine-tuning.ipynb
|_ training-ml-algorithms.ipynb
|_ training-muril-w-adapters.ipynb

• muril-fine-tuning.ipynb contains code for MuRIL Fine-tuning.
• training-ml-algorithms.ipynb contains code for classic ML algorithms train-eval.
• training-muril-w-adapters.ipynb fine-tunes MuRIL by injecting LoRA Adapters.

I train three algorithms: SVM, Random Forest, XGBoost. The Precision, Recall and F1 Score for all three are as follows:

SVM performs the best while Random Forest performs the worst. However, the Accuracy metrics for all three algorithms are disappointing.

The hyperparameters for full-model MuRIL Fine-tuning are:

batch_size = 32
learning_rate = 2e-5
epochs = 4

The process:

• Use google/muril-base-cased from HuggingFace Transformers library along with AutoModelForSequenceClassification
• Divide the dataset into Train and Validation sets
• Perform a 90:10 split
• Tokenize data and pass it to HuggingFace Trainer
• Train the model with Task 1 Dataset
• Evaluate the trained model with Holdout Testing Set (Task 1)
• Train again with Train set (Task 2)
• Evaluate the trained model with Holdout Testing Set (Task 1 and Task 2)
• Note the performance

The hyperparameters for MuRIL with LoRA Adapters fine-tuning are:

batch_size = 32
learning_rate = 2e-5
epochs = 20            # with early-stopping callback

The process:

• Use google/muril-base-cased from HuggingFace Transformers library along with AutoModelForSequenceClassification
• Inject PEFT LoRA adapters into the base model
• Divide the dataset (Task 1 and Task 2) into Train and Validation sets
• Perform a 90:10 split
• Tokenize data and pass it to HuggingFace Trainer
• Train the model with Adapter 1
• Evaluate the trained model with Holdout Testing Set (Task 1)
• Activate Adapter 2 and train again with Train set (Task 2)
• Evaluate the trained model with Holdout Testing Set (Task 1 and Task 2)
• Note the performance

The F1 Score for both full-model MuRIL fine-tuning and MuRIL with Adapters is as follows:

Conclusion: Although full-model Fine-tuning performs well for Task 1, its performance drops after continual learning. On the other hand, MuRIL (with adapters) performs well for Task 1 both before and after continual learning.