RLHF and Alignment - How ChatGPT Learns Human Values

Public lesson preview

Understand how raw language models are transformed into helpful, harmless assistants through supervised fine-tuning, reward modelling, and reinforcement learning from human feedback.

Learning objectives

By the end of this lesson, you will understand:

• ✓The Alignment Problem
• ✓Stage 1: Supervised Fine-Tuning (SFT)
• ✓Stage 2: Reward Model Training
• ✓Stage 3: Reinforcement Learning (PPO)
• ✓DPO: Cutting Out the Middleman

Lesson outline

1. The Alignment Problem
2. Stage 1: Supervised Fine-Tuning (SFT)
3. Stage 2: Reward Model Training
4. Stage 3: Reinforcement Learning (PPO)
5. DPO: Cutting Out the Middleman
6. Constitutional AI: Self-Supervision at Scale
7. Red Teaming
8. The Alignment Tax
9. 📚 Further Reading