Sign in to confirm you’re not a bot
This helps protect our community. Learn more

Let's reproduce GPT-2 (124M)

Andrej Karpathy
837K subscribers
<__slot-el>
<__slot-el>
790K views 10 months ago
We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Links:  …...more
...more

intro: Let’s reproduce GPT-2 (124M)

0:00

exploring the GPT-2 (124M) OpenAI checkpoint

3:39

SECTION 1: implementing the GPT-2 nn.Module

13:47

loading the huggingface/GPT-2 parameters

28:08

implementing the forward pass to get logits

31:00

sampling init, prefix tokens, tokenization

33:31

sampling loop

37:02

sample, auto-detect the device

41:47

let’s train: data batches (B,T) → logits (B,T,C)

45:50

cross entropy loss

52:53

optimization loop: overfit a single batch

56:42

data loader lite

1:02:00

parameter sharing wte and lm_head

1:06:14

model initialization: std 0.02, residual init

1:13:47

SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms

1:22:18

Tensor Cores, timing the code, TF32 precision, 333ms

1:28:14

float16, gradient scalers, bfloat16, 300ms

1:39:38

torch.compile, Python overhead, kernel fusion, 130ms

1:48:15

flash attention, 96ms

2:00:18

nice/ugly numbers. vocab size 50257 → 50304, 93ms

2:06:54

SECTION 3: hyperpamaters, AdamW, gradient clipping

2:14:55

learning rate scheduler: warmup + cosine decay

2:21:06

batch size schedule, weight decay, FusedAdamW, 90ms

2:26:21

gradient accumulation

2:34:09

distributed data parallel (DDP)

2:46:52

datasets used in GPT-2, GPT-3, FineWeb (EDU)

3:10:21

validation data split, validation loss, sampling revive

3:23:10

evaluation: HellaSwag, starting the run

3:28:23

SECTION 4: results in the morning! GPT-2, GPT-3 repro

3:43:05

shoutout to llm.c, equivalent but faster code in raw C/CUDA

3:56:21

summary, phew, build-nanogpt github repo

3:59:39
Let's reproduce GPT-2 (124M)
25KLikes
790,687Views
Jun 92024
We reproduce the GPT-2 (124M) from scratch. This video covers the whole process: First we build the GPT-2 network, then we optimize its training to be really fast, then we set up the training run following the GPT-2 and GPT-3 paper and their hyperparameters, then we hit run, and come back the next morning to see our results, and enjoy some amusing model generations. Keep in mind that in some places this video builds on the knowledge from earlier videos in the Zero to Hero Playlist (see my channel). You could also see this video as building my nanoGPT repo, which by the end is about 90% similar. Links: Supplementary links: Chapters: 00:00:00 intro: Let’s reproduce GPT-2 (124M) 00:03:39 exploring the GPT-2 (124M) OpenAI checkpoint 00:13:47 SECTION 1: implementing the GPT-2 nn.Module 00:28:08 loading the huggingface/GPT-2 parameters 00:31:00 implementing the forward pass to get logits 00:33:31 sampling init, prefix tokens, tokenization 00:37:02 sampling loop 00:41:47 sample, auto-detect the device 00:45:50 let’s train: data batches (B,T) → logits (B,T,C) 00:52:53 cross entropy loss 00:56:42 optimization loop: overfit a single batch 01:02:00 data loader lite 01:06:14 parameter sharing wte and lm_head 01:13:47 model initialization: std 0.02, residual init 01:22:18 SECTION 2: Let’s make it fast. GPUs, mixed precision, 1000ms 01:28:14 Tensor Cores, timing the code, TF32 precision, 333ms 01:39:38 float16, gradient scalers, bfloat16, 300ms 01:48:15 torch.compile, Python overhead, kernel fusion, 130ms 02:00:18 flash attention, 96ms 02:06:54 nice/ugly numbers. vocab size 50257 → 50304, 93ms 02:14:55 SECTION 3: hyperpamaters, AdamW, gradient clipping 02:21:06 learning rate scheduler: warmup + cosine decay 02:26:21 batch size schedule, weight decay, FusedAdamW, 90ms 02:34:09 gradient accumulation 02:46:52 distributed data parallel (DDP) 03:10:21 datasets used in GPT-2, GPT-3, FineWeb (EDU) 03:23:10 validation data split, validation loss, sampling revive 03:28:23 evaluation: HellaSwag, starting the run 03:43:05 SECTION 4: results in the morning! GPT-2, GPT-3 repro 03:56:21 shoutout to llm.c, equivalent but faster code in raw C/CUDA 03:59:39 summary, phew, build-nanogpt github repo Corrections: I will post all errata and followups to the build-nanogpt GitHub repo (link above) SuperThanks: I experimentally enabled them on my channel yesterday. Totally optional and only use if rich. All revenue goes to to supporting my work in AI + Education.

Follow along using the transcript.

Andrej Karpathy

837K subscribers