Skip to content

Llama 3

Llama 3 (Grattafiori et al., 2024) is a decoder-only transformer language model developed by Meta. It stacks pre-norm blocks with GQA self-attention and SwiGLU feed-forward layers. It uses RMSNorm for normalization and RoPE for positional information, with no bias. The model predicts the next token by computing a probability distribution over a vocabulary of 128,256 tokens.

References: Paper, Checkpoint (HuggingFace), Repository

Configuration

namespace llama3 = model::production::llama3;

auto modelCfg = llama3::ModelConfig::_3_2_1B();

llama3::Net net({
    .device        = device,
    .wDefaultDtype = Dtype::Fp16,
    .oDefaultDtype = Dtype::Fp32,
}, modelCfg, {
    .maxRuntimeContext = 4096,
});

NeuralNetConfig

Llama 3 uses the defaults; the loader reads each RmsNorm's scale at fp32. For details, see NeuralNet.

ModelConfig

Factory variant embedDim ffnHiddenDim numQHeads numKVHeads nBlocks
_3_2_1B() "3.2-1B" 2048 8192 32 8 16
_3_2_3B() "3.2-3B" 3072 8192 24 8 28

Both sizes use vocabSize 128,256 and maxTrainedContext 131,072. They also share the same ropeCfg value (rope(32.0f)). The factor of 32 is the RoPE frequency scaling factor for the Llama 3 family.

RuntimeConfig

This determines the dtype and the amount of KV cache to allocate, and the unit in which a prompt is fed. If maxRuntimeContext is omitted, it uses maxTrainedContext; specifying a value allocates only that amount to save memory. Allocating the full trained context of 131,072 requires too much memory, so the demo and benchmarks reduce it to 4,096 (the same value as n_ctx in the llama.cpp benchmark).

Field Default
kvCacheDtype Dtype::Fp16
maxRuntimeContext uint32_t(-1): up to maxTrainedContext
prefillChunk 0: the whole prompt in one pass

vAi graph

In vAi, a model is a DAG built by connecting nodes with -. For details, see Node and NodeGroup.

Tq: number of input tokens, D: embedding dimension, H: number of query heads, Hkv: number of KV heads.

Model architecture

input("inTokenIds") - embedding - decoder - lmHead - topkSampling - output("nextTokenId");

decoder is the shared Transformer group. It connects the blocks in order and ends with finalNorm.

for (size_t i = 1; i < blocks.size(); ++i)
    blocks[i - 1] - blocks[i];
blocks.back() - finalNorm;

NodeGroup

Llama 3 uses these node groups. All of them are shared groups. Transformer stacks TransformerBlock groups, and each TransformerBlock contains nested MultiHeadAttention and GatedFeedForward groups.

TransformerBlock

input - norm1 -    mhAttn - lastRow - gate/"then" - norm2 -    ffn;
input - "residual"/mhAttn - lastRow - gate/"then" - "residual"/ffn;

The final projection in each sublayer accepts the residual through acceptResidual and adds it within the kernel, without a separate Add. lastRow and gate are optional members created by the configuration, and an absent one drops out of the connection. Llama 3 creates lastRow only in the last block, so only the last row, which determines the next token, reaches the FFN. gate is created in the last block when prefillChunk is set; it closes for a chunk that produces no token and skips the computation from the FFN on.

MultiHeadAttention

With GQA, K and V are projected into fewer heads than the queries (H == 32, Hkv == 8). RoPE applies only to Q and K; V passes through unchanged.

input - projQ - splitQ - normQ - perQ - ropeQ - "Q"/attn;
input - projK - splitK - normK - perK - ropeK - "K"/attn;
input - projV - splitV         - perV         - "V"/attn - mergeOut - projOut;

normQ and normK are optional members created only for models with QK normalization, so Llama 3 does not create them.

With acceptResidual = true, projOut receives the block's residual through residual and adds it to the projection result.

Rope shares the KV cache token counter, so rotation uses the cumulative absolute position even during decode steps, rather than the offset within the current forward pass.

GatedFeedForward

Since gate and value read the same input and differ only in their weights, a single Linear computes both at twice the width. GatedMul applies SiLU to the first half and multiplies it by the second half.

gateValue - gatedMul - out;   // 2048 -> 16384 -> 8192 -> 2048

out also sets acceptResidual = true and receives the block's residual through residual.

Nodes used

TokenEmbedding, RmsNorm, Linear, Reshape, Permute, Rope, Attention, GatedMul, Identity, Slice, TopKSampling

Checkpoints

Place checkpoints in size-specific folders (ModelConfig::variant) under data/ckpts/llama3/meta/, with the tokenizer in data/tokenizers/llama3/. Both sizes share tokenizer.json.

data/
├── ckpts/llama3/meta/
│   ├── 3.2-1B/
│   │   └── model.safetensors
│   ├── 3.2-3B/
│   │   └── model.safetensors
│   └── ...
└── tokenizers/llama3/
    └── tokenizer.json

Download the files from the corresponding HuggingFace repository and use them without conversion. The weights are subject to the Llama 3.2 Community License.

mkdir -p data/ckpts/llama3/meta/3.2-1B data/tokenizers/llama3
curl -L -o data/tokenizers/llama3/tokenizer.json https://huggingface.co/unsloth/Llama-3.2-1B-Instruct/resolve/main/tokenizer.json
curl -L -o data/ckpts/llama3/meta/3.2-1B/model.safetensors \
  https://huggingface.co/unsloth/Llama-3.2-1B-Instruct/resolve/main/model.safetensors

Download model.safetensors and tokenizer.json directly from the HuggingFace repository's Files and versions tab or use the commands above.

The original meta-llama/* repositories are gated and require approval and a token. The commands above use an unsloth mirror containing the same tensors. The same license applies to mirror downloads. An Instruct checkpoint trained on chat tokens is required.

Loading weights

The code that maps checkpoint tensors to graph slots is in models/production/llama3/meta-weights.h. Loading takes one line at the call site. The specialization constructs the path (VAI_DATA_DIR "/ckpts/llama3/meta/" + modelCfg.variant), so only the size needs to be selected. For distribution tags, see distribution-weights.h in Writing custom models.

loadWeights<"meta">(net);

Running

Directory structure

models/production/llama3/
├── net.h                         Model and NodeGroup definitions
├── meta-weights.h                Meta (HuggingFace `meta-llama/*`) checkpoint mapping
├── demo.cpp                      demo-llama3, complete one prompt and measure timing
└── CMakeLists.txt

examples/chat-llama3/
├── main.cpp                      chat-llama3: multi-turn conversation
└── CMakeLists.txt

data/
├── ckpts/llama3/meta/
│   └── <variant>/                    ModelConfig::variant (3.2-1B, 3.2-3B)
│       └── model.safetensors
└── tokenizers/llama3/
    └── tokenizer.json

Build and run

cmake -B build -DVAI_BUILD_EXAMPLES=ON
cmake --build build --config RelWithDebInfo

demo-llama3

./build/bin/demo-llama3

The demo generates text that continues the text in prompt in demo.cpp. It measures prefill and decode times separately and prints them with the generated text.

chat-llama3

./build/bin/chat-llama3

This runs a multi-turn conversation from main.cpp. Conversation history stays in the KV cache, and the session ends on an empty line or when the cache is full.