docs: Rewrite PEFT integration guide with comprehensive examples

[behroozazarkhalili]

Resolves #4376

This PR completely rewrites the PEFT integration documentation to address the concerns raised in #4376.

Changes

• Comprehensive Trainer Coverage: Added detailed examples for SFT, DPO, and GRPO trainers with PEFT
• QLoRA Section: Added complete QLoRA guide with 4-bit and 8-bit quantization examples
• Prompt Tuning: Added new section on prompt tuning with configuration examples
• Updated Examples: All code examples updated to match current TRL API (LoRA r=32, alpha=16)
• Removed Outdated Content: Removed PPO-only focus and outdated information
• Enhanced Documentation: Added troubleshooting section, multi-GPU training guidance, and command-line arguments reference

Documentation Structure

1. Installation
2. Quick Start
3. PEFT with Different Trainers (SFT, DPO, GRPO)
4. QLoRA: Quantized Low-Rank Adaptation
5. Prompt Tuning
6. Advanced PEFT Configurations
7. Saving and Loading PEFT Models
8. Multi-GPU Training
9. Troubleshooting
10. Resources

All examples have been verified against the current TRL codebase and official scripts.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[sergiopaniego]

Thanks for the update!! Super detailed 😄

[sergiopaniego]

You can add somewhere a link to this example notebook: https://github.com/huggingface/trl/blob/main/examples/notebooks/sft_trl_lora_qlora.ipynb

[sergiopaniego]

We could leverage the usage of

<hfoptions id="command_line">
<hfoption id="SFT">
...
</hfoption>
<hfoption id="DPO">
...
</hfoption>
</hfoptions>

in this section to reduce the number of sections and improve readability.

[sergiopaniego]

We could focus only on the ideas needed for PEFT and simplify the rest to reduce the snippets.

For example, we could do:

training_args = SFTConfig(
   ...
)

similar for any part that is not strictly needed for the configuration

[sergiopaniego]

We could include here TRL notebooks, TRL examples, and recipes from cookbook (https://huggingface.co/learn/cookbook/index) that leverage PEFT

[sergiopaniego]

We actually have 3 different ways of adding the peft config to the trainer:

1. We give the model_name to the Trainer and the peft_config
2. We give the model instance and at the peft_config
3. We give the peft_model to the trainer directly, preparing it outside, without passing peft_config to the trainer.

We could add these details somewhere.

[qgallouedec]

instead of subsections, I'd write it with

<hfoptions id="trainer">
<hfoption id="SFT">

```
# Code for SFT
```

</hfoption>
<hfoption id="DPO">

```
Code for DPO
```

</hfoption>
</hfoptions>

[qgallouedec]

In my opinion, it is very important that all examples on this page contain an explicit learning rate (corresponding to 10x the trainer's default learning rate). Even better would be a small section explaining why, with a link to https://thinkingmachines.ai/blog/lora/.

[sergiopaniego]

And this one https://huggingface.co/docs/trl/lora_without_regret!

[qgallouedec]

I don't think these "No PEFT" sections are necessary

[qgallouedec]

Most a these are not specific to peft, so I recommend removing this section, and add these elements in reducing_memory_usage.md or speeding_up_training.md (can be done in a follow-up PR)

[behroozazarkhalili]

Addressed Reviewer Feedback

Thank you for the detailed review! I've addressed all the comments:

✅ Completed Changes

1. Learning Rates - CRITICAL:

   • Added comprehensive "Learning Rate Considerations" section with table showing recommended rates for each trainer
   • Added explicit learning_rate parameters to all 6 Python code examples (SFT, DPO, GRPO, QLoRA, Prompt Tuning)
   • Included LoRA blog post and LoRA Without Regret documentation references

2. Three PEFT Configuration Methods:

   • Added new section documenting all three approaches:
     • CLI flags (--use_peft)
     • Passing peft_config to trainer (recommended)
     • Applying PEFT to model directly with get_peft_model (advanced)
   • Each method includes pros/cons and code examples

3. Resources Enhancement:

   • Reorganized Resources section with categories
   • Added TRL notebooks (SFT with LoRA/QLoRA)
   • Added TRL examples directory link
   • Added TRL Cookbook recipes link
   • Added LoRA Without Regret documentation reference

4. Code Simplification:

   • All Python examples now use ellipsis (...) for non-PEFT configuration
   • Focus maintained on PEFT-specific parameters

5. Removed Sections:

   • Removed 3 "Full Training (No PEFT)" subsections from SFT, DPO, and GRPO
   • Removed entire Troubleshooting section

6. Import Order:

   • Fixed import order in QLoRA example (standard library before third-party)
   • Properly grouped and ordered all imports

Already Addressed

• Notebook link, hfoptions tabs, SFTTrainer import (already present in initial PR)

All changes committed in cbe38d7.

[sergiopaniego]

@behroozazarkhalili could you review the conflicts? 😄

[sergiopaniego]

Is this section still true? I can't find references to ppo_adapter_name so I'd suggest reviewing it and removing it if outdated

[sergiopaniego]

Should we apply the change proposed in here @behroozazarkhalili? 😄

[behroozazarkhalili]

✅ Yes, already applied!

What I have done:

1. Removed Outdated ppo_adapter_name Section ✅

• Commit: eee9673 - "docs: remove outdated ppo_adapter_name section from PEFT guide"
• Action: Removed 9 lines documenting non-existent ppo_adapter_name parameter

[sergiopaniego]

thanks!