KTO refactoring

[albertvillanova]

Systematically refactor KTOTrainer to align with modern trainer architecture patterns, improving maintainability, reducing complexity, and adding missing features.

Problem

KTOTrainer currently has complex code with several architectural issues.

Refactoring Plan

1: Foundation & Infrastructure

Establish modern patterns without breaking functionality

• Modernize Model Initialization: Refactor KTO [1/N]: Modernize model initialization #4783
  • Use create_model_from_path() helper
  • Simplify dtype and device_map handling
  • Clean up ref_model initialization
• Improve Config Validation: Refactor KTO [2/N]: Improve config validation in KTOConfig #4787

2: Dataset Processing Modernization

Adopt standard trainers' clean preprocessing patterns

• Extract _prepare_dataset method: Refactor KTO [3/N]: Extract dataset processing to _prepare_dataset method #4788

3: Reference Model Handling

Simplify ref model management

4: Loss Computation & Forward Pass

Reduce duplication in loss computation

5: Metrics & Logging

Match standard metrics system

6: Evaluation & Generation

Modern evaluation features

7: Documentation

Complete, clear documentation

Phase 8: Advanced Features

Feature parity with standard trainers

Phase 9: Testing

Production ready

[albertvillanova]

After internal discussion with @qgallouedec, we are coordinating to refactor both DPO and KTO. See related:

• [WIP] Refactor DPO #3906

WIP: Features to Remove

First, remove some features. Benefits:

• Less code to refactor
• Simpler architecture
• Easier DPO alignment
• Clearer core functionality
• Faster refactoring

A. Remove Encoder-Decoder Support

PR:

• Refactor KTO coordinated with DPO [a/N]: Remove encoder-decoder support #4792

Why Remove:

• ❌ Adds ~200-300 lines of conditional logic
• ❌ Complicates every dataset processing step
• ❌ Rarely used (most preference learning is causal LM)
• ❌ Different tokenization, label handling, model kwargs
• ❌ Testing burden (2x test coverage needed)

Already broken, minimal user impact.

B. Separate Prompt/Completion Truncation

Why Remove:

• ❌ Adds ~80 lines of complex truncation logic
• ❌ Confusing UX (users don't know which to set)
• ❌ Most users just want max_length
• ❌ truncation_mode is redundant with tokenizer settings

Replace with simple max_length only.