Llama 3¶
Llama 3 (Grattafiori et al., 2024) is a decoder-only transformer language model developed by Meta. It stacks pre-norm blocks with GQA self-attention and SwiGLU feed-forward layers. It uses RMSNorm for normalization and RoPE for positional information, with no bias. The model predicts the next token by computing a probability distribution over a vocabulary of 128,256 tokens.
References: Paper, Checkpoint (HuggingFace), Repository
Configuration¶
namespace llama3 = model::production::llama3;
auto modelCfg = llama3::ModelConfig::_3_2_1B();
llama3::Net net({
.device = device,
.wDefaultDtype = Dtype::Fp16,
.oDefaultDtype = Dtype::Fp32,
}, modelCfg, {
.maxRuntimeContext = 4096,
});
NeuralNetConfig¶
Llama 3 uses the defaults; the loader reads each RmsNorm's scale at fp32.
For details, see NeuralNet.
ModelConfig¶
| Factory | variant |
embedDim |
ffnHiddenDim |
numQHeads |
numKVHeads |
nBlocks |
|---|---|---|---|---|---|---|
_3_2_1B() |
"3.2-1B" |
2048 | 8192 | 32 | 8 | 16 |
_3_2_3B() |
"3.2-3B" |
3072 | 8192 | 24 | 8 | 28 |
Both sizes use vocabSize 128,256 and maxTrainedContext 131,072. They also share the same ropeCfg value (rope(32.0f)). The factor of 32 is the RoPE frequency scaling factor for the Llama 3 family.
RuntimeConfig¶
This determines the dtype and the amount of KV cache to allocate, and the unit in which a prompt is fed. If maxRuntimeContext is omitted, it uses maxTrainedContext; specifying a value allocates only that amount to save memory. Allocating the full trained context of 131,072 requires too much memory, so the demo and benchmarks reduce it to 4,096 (the same value as n_ctx in the llama.cpp benchmark).
| Field | Default |
|---|---|
kvCacheDtype |
Dtype::Fp16 |
maxRuntimeContext |
uint32_t(-1): up to maxTrainedContext |
prefillChunk |
0: the whole prompt in one pass |
vAi graph¶
In vAi, a model is a DAG built by connecting nodes with -. For details, see Node and NodeGroup.
Tq: number of input tokens, D: embedding dimension, H: number of query heads, Hkv: number of KV heads.
Model architecture¶
decoder is the shared Transformer group. It connects the blocks in order and ends with finalNorm.
NodeGroup¶
Llama 3 uses these node groups. All of them are shared groups.
Transformer stacks TransformerBlock groups, and each TransformerBlock contains nested MultiHeadAttention and GatedFeedForward groups.
TransformerBlock¶
input - norm1 - mhAttn - lastRow - gate/"then" - norm2 - ffn;
input - "residual"/mhAttn - lastRow - gate/"then" - "residual"/ffn;
The final projection in each sublayer accepts the residual through acceptResidual and adds it within the kernel, without a separate Add.
lastRow and gate are optional members created by the configuration, and an absent one drops out of the connection. Llama 3 creates lastRow only in the last block, so only the last row, which determines the next token, reaches the FFN. gate is created in the last block when prefillChunk is set; it closes for a chunk that produces no token and skips the computation from the FFN on.
MultiHeadAttention¶
With GQA, K and V are projected into fewer heads than the queries (H == 32, Hkv == 8). RoPE applies only to Q and K; V passes through unchanged.
input - projQ - splitQ - normQ - perQ - ropeQ - "Q"/attn;
input - projK - splitK - normK - perK - ropeK - "K"/attn;
input - projV - splitV - perV - "V"/attn - mergeOut - projOut;
normQ and normK are optional members created only for models with QK normalization, so Llama 3 does not create them.
With acceptResidual = true, projOut receives the block's residual through residual and adds it to the projection result.
Rope shares the KV cache token counter, so rotation uses the cumulative absolute position even during decode steps, rather than the offset within the current forward pass.
GatedFeedForward¶
Since gate and value read the same input and differ only in their weights, a single Linear computes both at twice the width. GatedMul applies SiLU to the first half and multiplies it by the second half.
out also sets acceptResidual = true and receives the block's residual through residual.
Nodes used¶
TokenEmbedding,
RmsNorm,
Linear,
Reshape,
Permute,
Rope,
Attention,
GatedMul,
Identity,
Slice,
TopKSampling
Checkpoints¶
Place checkpoints in size-specific folders (ModelConfig::variant) under data/ckpts/llama3/meta/, with the tokenizer in data/tokenizers/llama3/.
Both sizes share tokenizer.json.
data/
├── ckpts/llama3/meta/
│ ├── 3.2-1B/
│ │ └── model.safetensors
│ ├── 3.2-3B/
│ │ └── model.safetensors
│ └── ...
└── tokenizers/llama3/
└── tokenizer.json
Download the files from the corresponding HuggingFace repository and use them without conversion. The weights are subject to the Llama 3.2 Community License.
mkdir -p data/ckpts/llama3/meta/3.2-1B data/tokenizers/llama3
curl -L -o data/tokenizers/llama3/tokenizer.json https://huggingface.co/unsloth/Llama-3.2-1B-Instruct/resolve/main/tokenizer.json
curl -L -o data/ckpts/llama3/meta/3.2-1B/model.safetensors \
https://huggingface.co/unsloth/Llama-3.2-1B-Instruct/resolve/main/model.safetensors
Download model.safetensors and tokenizer.json directly from the HuggingFace repository's Files and versions tab or use the commands above.
The original
meta-llama/*repositories are gated and require approval and a token. The commands above use an unsloth mirror containing the same tensors. The same license applies to mirror downloads. An Instruct checkpoint trained on chat tokens is required.
Loading weights¶
The code that maps checkpoint tensors to graph slots is in models/production/llama3/meta-weights.h.
Loading takes one line at the call site. The specialization constructs the path (VAI_DATA_DIR "/ckpts/llama3/meta/" + modelCfg.variant), so only the size needs to be selected. For distribution tags, see distribution-weights.h in Writing custom models.
Running¶
Directory structure¶
models/production/llama3/
├── net.h Model and NodeGroup definitions
├── meta-weights.h Meta (HuggingFace `meta-llama/*`) checkpoint mapping
├── demo.cpp demo-llama3, complete one prompt and measure timing
└── CMakeLists.txt
examples/chat-llama3/
├── main.cpp chat-llama3: multi-turn conversation
└── CMakeLists.txt
data/
├── ckpts/llama3/meta/
│ └── <variant>/ ModelConfig::variant (3.2-1B, 3.2-3B)
│ └── model.safetensors
└── tokenizers/llama3/
└── tokenizer.json
Build and run¶
demo-llama3¶
The demo generates text that continues the text in prompt in demo.cpp. It measures prefill and decode times separately and prints them with the generated text.
chat-llama3¶
This runs a multi-turn conversation from main.cpp. Conversation history stays in the KV cache, and the session ends on an empty line or when the cache is full.