GPT-2¶
GPT-2 (Radford et al., 2019) is a decoder-only transformer language model developed by OpenAI. It stacks pre-norm blocks with causal self-attention and GELU feed-forward layers. The model predicts the next token by computing a probability distribution over a vocabulary of 50,257 tokens.
References: Paper, Checkpoint (HuggingFace), Repository
Configuration¶
namespace gpt2 = model::production::gpt2;
auto modelCfg = gpt2::ModelConfig::_124M();
gpt2::Net net({
.device = device,
.wDefaultDtype = Dtype::Fp16,
.oDefaultDtype = Dtype::Fp32,
}, modelCfg);
NeuralNetConfig¶
GPT-2 uses the defaults; the loader reads each LayerNorm's coefficients at fp32.
For details, see NeuralNet.
ModelConfig¶
| Factory | variant |
embedDim |
numQHeads |
nBlocks |
|---|---|---|---|---|
_124M() |
"124M" |
768 | 12 | 12 |
_355M() |
"355M" |
1024 | 16 | 24 |
_774M() |
"774M" |
1280 | 20 | 36 |
_1_5B() |
"1.5B" |
1600 | 25 | 48 |
All sizes use vocabSize 50,257 and maxTrainedContext 1,024.
RuntimeConfig¶
This determines the dtype and the amount of KV cache to allocate. If maxRuntimeContext is omitted, it uses maxTrainedContext; specifying a value allocates only that amount to save memory.
| Field | Default |
|---|---|
kvCacheDtype |
Dtype::Fp16 |
maxRuntimeContext |
uint32_t(-1): up to maxTrainedContext |
vAi graph¶
In vAi, a model is a DAG built by connecting nodes with -. For details, see Node and NodeGroup.
Tq: number of input tokens, D: embedding dimension, H: number of heads.
Model architecture¶
decoder is the shared Transformer group. It connects the blocks in order and ends with finalNorm.
NodeGroup¶
GPT-2 uses these node groups. All of them are shared groups.
Transformer stacks TransformerBlock groups, and each TransformerBlock contains nested MultiHeadAttention and FeedForward groups.
TransformerBlock¶
input - norm1 - mhAttn - lastRow - gate/"then" - norm2 - ffn;
input - "residual"/mhAttn - lastRow - gate/"then" - "residual"/ffn;
The final projection in each sublayer accepts the residual through acceptResidual and adds it within the kernel, without a separate Add.
lastRow and gate are optional members created by the configuration, and an absent one drops out of the connection. GPT-2 creates lastRow only in the last block, so only the last row, which determines the next token, reaches the FFN.
MultiHeadAttention¶
Q, K, and V are computed with a single {D, 3D} projection, then split into three slices along the head axis.
projQKV - splitHeads - perQKV - sliceQ - normQ - ropeQ - "Q"/attn;
perQKV - sliceK - normK - ropeK - "K"/attn;
perQKV - sliceV - "V"/attn - mergeOut - projOut;
normQ, normK, ropeQ, and ropeK are optional members too. GPT-2 uses neither QK normalization nor RoPE, so none of the four is created.
With acceptResidual = true, projOut receives the block's residual through residual and adds it to the projection result.
FeedForward¶
This group connects two Linear nodes. GELU is attached as an epilogue to the first Linear and runs within GEMM, without a separate node.
fc2 also sets acceptResidual = true and receives the block's residual through residual.
Nodes used¶
TokenEmbedding,
LayerNorm,
Linear,
Reshape,
Permute,
Attention,
Identity,
Slice,
TopKSampling
Checkpoints¶
Place checkpoints in size-specific folders (ModelConfig::variant) under data/ckpts/gpt2/openai/, with the tokenizer in data/tokenizers/gpt2/.
All four sizes share tokenizer.json.
data/
├── ckpts/gpt2/openai/
│ ├── 124M/
│ │ └── model.safetensors
│ ├── 355M/
│ │ └── model.safetensors
│ └── ...
└── tokenizers/gpt2/
└── tokenizer.json
Download the files from the corresponding HuggingFace repository and use them without conversion. The weights are licensed under MIT.
mkdir -p data/ckpts/gpt2/openai/124M data/tokenizers/gpt2
curl -L -o data/tokenizers/gpt2/tokenizer.json https://huggingface.co/openai-community/gpt2/resolve/main/tokenizer.json
curl -L -o data/ckpts/gpt2/openai/124M/model.safetensors \
https://huggingface.co/openai-community/gpt2/resolve/main/model.safetensors
Download model.safetensors and tokenizer.json directly from the HuggingFace repository's Files and versions tab or use the commands above.
Loading weights¶
The code that maps checkpoint tensors to graph slots is in models/production/gpt2/openai-weights.h.
Loading takes one line at the call site. The specialization constructs the path (VAI_DATA_DIR "/ckpts/gpt2/openai/" + modelCfg.variant),
so only the size needs to be selected. For distribution tags, see distribution-weights.h in Writing custom models.
Running¶
Directory structure¶
models/production/gpt2/
├── net.h Model and NodeGroup definitions
├── openai-weights.h OpenAI (HuggingFace `openai-community/*`) checkpoint mapping
├── demo.cpp demo-gpt2: complete one prompt and measure timing
└── CMakeLists.txt
data/
├── ckpts/gpt2/openai/
│ └── <variant>/ ModelConfig::variant: 124M, 355M, 774M, 1.5B
│ └── model.safetensors
└── tokenizers/gpt2/
└── tokenizer.json
Build and run¶
The demo generates text that continues the text in prompt. It measures prefill and decode times separately and prints them with the generated text.