Sign in
Advanced Training Mechanics

In-Context Learning

How models learn from examples in their prompt without weight updates

What it is

In-context learning (ICL) is the ability of an LLM to adapt its behavior based on examples provided in the prompt, without any gradient updates to its weights. Provide 5 examples of a new classification task, and the model generalizes to new instances, even for tasks it's never seen during training.

This is fundamentally different from traditional machine learning, where adaptation requires re-training. ICL appears to be an emergent capability of sufficiently large models, smaller models don't show robust in-context learning.

The mechanism is still actively studied. Leading theories suggest models perform a form of implicit gradient descent in the forward pass, or learn general pattern-matching algorithms during training that apply at inference time.

Why it matters

ICL is why few-shot prompting works and why you can teach a model new tasks through careful prompt construction. Understanding ICL helps you use few-shot prompting more effectively and understand why structured prompt design sometimes produces dramatically better results than natural language instructions alone.

Related concepts

Resources

Prompt Engineering
lilianweng.github.io· Weng's comprehensive survey covers zero-shot, few-shot, and in-context learning with academic rigor but clear writing. Includes discussion of why example ordering matters, calibration techniques, and connections to chain-of-thought. The definitive blog post on this topic.
25 min
In-Context Learning Guide
prompthub.us· More practical and beginner-friendly than Weng's post. Covers how ICL differs from few-shot prompting, whether models "really learn" in context, and includes templates. Good balance of theory and application.
12 min
What is In-Context Learning, and How Does It Work?
lakera.ai· Clear beginner-level explainer with good diagrams. Covers the mechanistic intuition (transient learning, no weight updates), chain-of-thought prompting as extension, and real-world applications.
10 min
Few-Shot Prompting
learnprompting.org· Hands-on tutorial format with examples. Good for recruits who want to try ICL themselves. Connects zero-shot → one-shot → few-shot clearly.
8 min
What Is In-Context Learning in Deep Learning?
youtube.com· Dedicated explainer on in-context learning covering the mechanistic intuition and why it's surprising. **Confirmed.**
20 min