Name	Name	Last commit message	Last commit date
parent directory ..
README.md	README.md
metrics.py	metrics.py
mezo.sh	mezo.sh
run.py	run.py
tasks.py	tasks.py
templates.py	templates.py
utils.py	utils.py

Name

Last commit message

Last commit date

Example: Apply MeZO on LLMs

Modified from MeZO

Usage

Use run.py for all functions (zero-shot/MeZO):

python run.py {ARGUMENTS}

Please read run.py for a complete list of arguments. We introduce some of the most important ones below.

--num_train: Number of training examples.
--num_dev: Number of validation examples.
--num_test: Number of testing examples.
--model_name: HuggingFace model name or path.
--task_name: Task name.
--trainer: can be none (zero-shot) or zo (MeZO).
--train_as_classification: turn this on for classification tasks (Cross Entropy over likelihood of each class' label words). Otherwise it is LM-style teacher forcing.
--zo_eps: MeZO hyperparameter epsilon.
--zo_method: choose zeroth-order methods.
--zo_mode: can be zo (on device) or zo2 (offloading).
--offloading_device: offloading device.
--working_device: main working device.

Example:

# MeZO (full-parameter fine-tuning)
MODEL=facebook/opt-2.7b TASK=SST2 MODE=ft LR=1e-7 EPS=1e-3 STEPS=20000 EVAL_STEPS=4000 bash mezo.sh