Reference model for LoRA

[SpaceHunterInf]

Greetings,

Quick question about our LoRA setup: are we initializing a separate reference model during training (e.g., for KL/anchor comparisons), or are we reusing the frozen base model as the reference since the base is frozen under PEFT?

If a separate reference is currently spun up, is there a way to configure the pipeline to reuse the base model as the reference to save memory/compute? Pointers to the relevant config flags or code path would be much appreciated.

Thank you very much.

[pramodith]

@SpaceHunterInf I believe this is already in place. In GRPO for e.g. If you're using a PEFT model a separate reference model isn't initialized, as seen

trl/trl/trainer/grpo_trainer.py

Lines 458 to 461 in 47b1aa7

	elif is_peft_model(model):
	# If PEFT is used, the reference model is not needed since the adapter can be disabled
	# to revert to the initial model.
	self.ref_model = None

and when self.beta>0 (i.e. we have a KL penalty in our loss) we disable the adapter and get the logits from the frozen model as you suggested, as seen in the code here

trl/trl/trainer/grpo_trainer.py

Line 1495 in 47b1aa7

with self.accelerator.unwrap_model(self.model).disable_adapter():

.

So you shouldn't need to add any new configs 😄 .