We use LLaMA-Factory to do sft training.

git clone --depth 1 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e ".[torch,metrics]" --no-build-isolation

We provide our training setting yaml file for your reference:

### model
model_name_or_path: /path/to/base/reward/model

### method
stage: sft
do_train: true
finetuning_type: full
deepspeed: examples/deepspeed/ds_z3_config.json
freeze_vision_tower: true
freeze_multi_modal_projector: false

### dataset
dataset:  # register training dataset following LLaMA-Factory tutorial
template: qwen2_vl
cutoff_len: 10000

overwrite_cache: true
preprocessing_num_workers: 16

### output
output_dir: /path/to/output/dir
logging_steps: 10
save_steps: 400
plot_loss: true
overwrite_output_dir: true

### train
per_device_train_batch_size: 4
gradient_accumulation_steps: 4
learning_rate: 5.0e-6
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000

### eval
val_size: 0.1
per_device_eval_batch_size: 1
eval_strategy: steps
eval_steps: 100

report_to: 'tensorboard' # or wandb

Can you provide a script for training a reward model? #7

Description

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions