Sign in
Model Types

Base Models

Pure autocomplete, powerful but raw and hard to direct

What it is

A base model is the direct output of pre-training, a model trained only to predict the next token with no instruction fine-tuning or safety training applied. Base models have internalized an enormous amount of knowledge and capability from their training data, but they express it inconsistently.

If you prompt a base model to translate text to Spanish, it might do the translation, or it might continue the text as if it were a forum thread, or generate a different translation task. It has the capability but can't reliably surface it on demand.

Base models are valuable for researchers who want to fine-tune models for specific tasks from a clean starting point, or who want to study what capabilities emerge purely from scale and data.

Why it matters

Understanding base models clarifies why post-training matters so much, and why "the model knows X" doesn't mean "the model will tell you X." It also gives context for evaluating open-weight model releases, when Meta releases a Llama base model vs. an instruct model, they serve very different purposes. A base model needs additional fine-tuning before being deployable in a product.

Related concepts

Resources

Deep Dive into LLMs like ChatGPT (Llama 3.1 base model inference)
youtube.com· Live demo of a base model (Llama 3.1) showing how it just completes text rather than answering questions. Demonstrates exactly why base models are "document completers," not assistants, and why post-training matters. The examples of raw Llama 3.1 base completions are eye-opening.
17 min
Deep Dive into LLMs like ChatGPT (Base model demos, later segment)
youtube.com· Additional base model demonstration later in the video, showing more examples of base model behavior in context. Useful companion to segment #1 above.
9 min
Let's Build GPT: from scratch (pre-training output section)
youtube.com· Shows a tiny base model generating Shakespeare-like text. Viscerally demonstrates what "next token prediction" produces before any fine-tuning.
10 min
Base LLM vs. Instruction-Tuned LLM
toloka.ai· Clear side-by-side comparison of base vs. instruction-tuned models. Covers training methods, capabilities, use cases, and when you'd use each. Good reference for recruits.
8 min
Foundation vs. Instruct vs. Thinking Models
blog.alexewerlof.com· Excellent mental model: base = library, instruct = app, thinking = operator. Shows the actual chat template markup that converts instruct models into conversations under the hood.
10 min
PreviousBeginning of section
NextInstruction-tuned Models