Learn AI by building it

Build a language model, one token at a time.

TokenLab teaches how large language models are really made: tokenizer, embeddings, attention, and sampling, by letting you assemble and run tiny models right on your phone.

Download for iPhone See how it works

4.9 ratingBuilt for engineers and CS students

Lesson 12

Attention

71%

Input

Thecatsat

next-token prediction

mat64%

floor22%

sofa9%

Build block

Self-Attention

QKV

From tokens to transformer

You learn every stage by building it

Tokenizer

Break text into tokens and map them to IDs, then back again.

Embeddings

Turn token IDs into vectors the model can actually reason over.

Attention

Let tokens weigh each other through Query, Key, and Value.

Sampling

Convert logits into the next token with temperature and top-k.

Bite-size lessons

Concepts that finally click

Short, precise lessons with the real math laid bare, then a quick check to make sure it stuck. Self-paced, no streaks, no pressure.

Real formulas, explained plainly

Inline quizzes with instant feedback

Pick up exactly where you left off

Chapter 3 / Lesson 3

Query, Key & Value

softmax(Q x K^T / sqrt(dk)) x V

Quick check

What does a token's Query represent?

What the token is looking for in others

The model's fixed vocabulary

Model pipeline

Tokenizerready

Embeddingsready

Self-Attentiondrag to place

+ MLP block

Interactive builders

Assemble a model block by block

Stack tokenizer, embeddings, and attention blocks into a working architecture. Tap any block to inspect its shapes, weights, and what flows through it.

Playground

A phone-sized micro-transformer lab

Choose a preset like Tiny GPT, Adapter, RAG, or Device Fit, then adjust the pieces that actually shape a small model: tokenizer, layers, embedding size, attention heads, context length, dataset size, training steps, eval split, temperature, top-p, and max tokens.

Each run shows estimated parameters, memory, latency, train loss, eval loss, pass rate, token flow, and pass/fail test results for training, alignment, RAG, or device readiness. Save runs as checkpoints, compare experiments, and keep the playground as your long-term model-building workspace.

Experiment 014

Tiny GPT playground

checkpoint saved

Tiny GPTAdapterRAGDevice Fit

Tokenizer

Layers

Embedding size

Attention heads

Context length

Dataset size

Training steps

Eval split

Temperature

Top-p

Max tokens

Run metrics

Params

1.8M

Memory

42 MB

Latency

18 ms

Train loss

1.72

Eval loss

1.91

Pass rate

86%

Test results

Trainingpass

Alignmentpass

RAGready

Device fitpass

token flow

clean text to tokens to logits to sampled answer

Run real inference

Watch the probabilities move

Feed your model a prompt and see the next-token distribution live. Drag the temperature and watch a confident model turn playful in real time.

Temperature0.80

The cat sat on the ___

next-token prediction

mat57%

floor24%

sofa11%

rug5%

table3%

18 core modules / 77 lessons

From first principles to embodied AI

Start with the mechanics of a micro LLM, then move through post-training, retrieval, agents, evaluation, on-device deployment, robotics, and release readiness.

Micro LLM Core

5 modules

Module 014 lessons

ML Foundations

Data, parameters, loss, gradients, validation, and generalization.

Module 024 lessons

Text to Tokens

Dataset preparation, tokenization, context windows, and next-token targets.

Module 038 lessons

Build From First Principles

Progress from count-based models to neural networks and causal transformers.

Module 044 lessons

Transformer Internals

Embeddings, attention, MLPs, residual connections, and normalization.

Module 054 lessons

Train the Micro LLM

Configuration, batching, optimization, checkpoints, sampling, and debugging.

Post-Training

3 modules

Module 064 lessons

Fine-Tuning and Adapters

Prompting versus retrieval, full fine-tuning, LoRA, adapters, and serving.

Module 075 lessons

Alignment Foundations

Instruction training, preference data, reward models, DPO, and PPO-style RLHF.

Module 084 lessons

Preference Optimization in Practice

Preference-data quality, reward calibration, DPO controls, and behavioral evaluation.

Retrieval and Agents

2 modules

Module 094 lessons

RAG and Embeddings

Embedding spaces, chunking, retrieval, reranking, and grounded generation.

Module 104 lessons

Reasoning, Tools, and Agents

Structured outputs, tool calling, routing, memory, state, and failure recovery.

Evaluation and Data

2 modules

Module 114 lessons

Data Quality and Synthetic Data

Licensing, privacy, deduplication, contamination, filtering, and synthetic examples.

Module 124 lessons

Evaluation and Observability

Golden evaluations, behavioral metrics, traces, regressions, and release gates.

On-Device

1 module

Module 134 lessons

On-Device Optimization

KV caching, distillation, quantization, profiling, memory, latency, and Apple deployment.

Embodied AI and Robotics

4 modules

Module 144 lessons

Embodied AI Foundations

Robot state, perception, control layers, feedback loops, and partial observability.

Module 155 lessons

Language, Vision, and Action

Scene grounding, spatial reasoning, skill planning, action representations, and VLA models.

Module 164 lessons

Robot Policy Learning

Demonstrations, behavior cloning, policy fine-tuning, rewards, and offline learning.

Module 172 lessons

Safe Robot Systems

Simulation, sim-to-real evaluation, independent safety limits, and human oversight.

Capstone

1 module

Module 185 lessons

Evaluate and Ship

Quality measurement, uncertainty, device constraints, privacy, model documentation, and release readiness.

Ready to build your first model?

Start with tokenization today. Free to begin, new lessons every month.

Download for iPhone