From scratch
We start from absolute zero and invent every key element of LLM architecture + training from scratch. Intuition for high-school-level math beneficial (general algebra, derivatives, vectors+matrices). Nothing else required.
⟨⟩
Chapters
First StepsFirst Steps
Text & TokenizationTokenization
N-grams & the CurseN-grams
Similarity & EmbeddingsEmbeddings
Function ApproximationCompression
Neural NetworksNetworks
MLE & Gradient DescentGradient Descent
SGD & BackpropBackprop
RegularizationRegularization
Milestone: Neural LMNeural LM
Batch NormBatch Norm
Adam & AdamWAdam & AdamW
BPEBPE
Residual ConnectionsResiduals
Milestone: Neural LM 2.0Neural LM 2.0
AttentionAttention
Transformer ArchitectureTransformer
RoPERoPE
RMS NormRMS Norm
Milestone: GPTGPT
SwiGLUSwiGLU
Mixtures of ExpertsMoE
Milestone: MoE-GPTMoE-GPT
Supervised Fine-Tuning (SFT)SFT
Reinforcement Learning (RL)RL
Rewards & Reward ModelsReward Models
Policy GradientsPolicy Gradients
Advantage EstimationAdvantages
PPO & KL PenaltyPPO
Milestone: RL FrameworkRL Framework
One coherent story
Simple approach → problem → analysis → less simple approach. Rinse and repeat. Urgent issues first, others tracked for downstream improvements.
Heavy on intuition
First-principle arguments, toy problems, information-oriented reasoning and specific examples wherever appropriate. No top-down explanations, no regurgitation of deprecated framings.
Visual and effective
Fancy visuals and interactive widgets where they actually aid understanding. Highlight boxes for clarity. Text for the main narrative.
Deliberately conceptual
Concepts, concepts, concepts. This is intended as an on-ramp to a genuine understanding of LLMs for people who want to dig deep. It is not a coding guide.