제공 노드들¶
빌트인 노드 28종. 개념은 노드(Node)를, 새 노드를 만드는 법은 커스텀 노드와 커널을 참조하세요.
목록¶
| 노드 | 요약 | 커널 |
|---|---|---|
| Activation | Applies an element-wise activation. | activation.comp |
| Add | element-wise addition for residual connections | add.comp |
| Argmax | Row-wise argmax / greedy sampling | argmax.comp |
| Attention | fused KV cache append + causal attention (GQA), split between four paths | attention-fa-intel.comp, attention-fa-cm-intel.comp, attention-fa-cm.comp, attention-fa.comp, attention-fa-reduce-intel.comp, attention-fa-reduce.comp, attention.comp |
| Concat | join inputs along one axis | concat.comp |
| Convolution | 2D convolution with device-selected direct, implicit, or Winograd plans | conv-k1s1-gemm-cm.comp, conv-k1s1-gemm.comp, conv-k3s2-gemm-cm.comp, conv-k3s1-implicit-gemm-cm.comp, conv-k3s1-implicit-gemm.comp, conv-implicit-gemm.comp, conv-implicit-gemm-cm.comp, conv-k3s2-gemm.comp, cm-transform.comp, conv-k3s1-winograd-filter-transform.comp, conv-k3s1-winograd-input.comp, conv-k3s1-winograd-output.comp, bn-reduce.comp, conv-k3s1-winograd-gemm.comp, conv-k3s1-winograd-gemm-cm.comp |
| CyclicTableReader | read successive table rows, wrapping at the end | cyclic-table-reader.comp |
| EleAffine | element-wise affine, O = scale * in0 + shift | ele-affine.comp |
| Flatten | Flatten tensor to 1D | — |
| Gate | conditional branch (no computation) | — |
| GatedMul | element-wise multiply with one operand activated (GLU gating) | gated-mul.comp |
| GroupNorm | group normalization over a CHW feature map | group-norm.comp |
| Identity | Pass-through node (no computation) | — |
| IndexCopy | write source slices at listed indices, as torch.index_copy | index-copy.comp |
| LayerNorm | layer normalization over the feature axis, single kernel | layer-norm.comp |
| Linear | O = I · Wᵀ, split between two output decompositions | linear-pack-intel.comp, linear-preshuffle.comp, linear-slab.comp, linear-tile-intel.comp, linear-cm-amd.comp, linear-cm-nv.comp, linear-cm.comp, linear-tile.comp, linear-reduce.comp |
| Mul | Element-wise multiplication | mul.comp |
| NearestUpsample | Replicates CHW pixels by an integer scale factor. | upsample.comp |
| Permute | Permutes the axes of a tensor of any rank. | permute.comp |
| Pool2d | 2D pooling with a square kernel (MaxPool2d, AvgPool2d) | pool2d.comp |
| Reshape | Reshape tensor to specified dimensions | — |
| RmsNorm | RMS normalization over the feature axis, single kernel | rms-norm.comp |
| Rope | rotary position embedding (half-split rotation) | rope.comp |
| RowSliceScatter | write slices into rows, as torch.slice_scatter on dim 0 | — |
| Slice | Take a run of consecutive indices along one axis | slice.comp |
| TokenEmbedding | token embedding, optionally fused with positional embedding | token-embedding.comp |
| TopKSampling | row-wise top-k sampling, one kernel run as two passes | topk-sampling.comp |
| TypeCast | Casts each tensor element to another dtype. | cast.comp |
Activation¶
Applies an element-wise activation.
시그니처
- 입력: in0 [any shape]
- 출력: epilogue(in0); optionally in-place
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
epilogue |
vai::Epilogue |
vai::Epilogue::None |
|
inPlace |
bool |
false |
동작 및 제약
The inPlace option aliases the output to the input and requires matching input and output dtypes. Otherwise, the output uses separate storage.
커널
vai/shaders/activation.comp
Add¶
element-wise addition for residual connections
소스: vai/nodes/add.h
시그니처
- 입력: in0, in1 [same shape]
- 출력: in0 + in1
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
in1 |
input | |
out0 |
output |
동작 및 제약
Inputs must have identical shapes and dtypes.
커널
vai/shaders/add.comp
Argmax¶
Row-wise argmax / greedy sampling
시그니처
- 입력: in0 [M, N] (fp32 logits)
- 출력: out0 [M] (one raw uint32 index per row, in 4-byte slots)
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
동작 및 제약
Each row is reduced independently. out0[m] is the index of the largest value in in0[m, :]. Output elements are raw uint32 values, not fp32 numeric values.
커널
vai/shaders/argmax.comp
Attention¶
fused KV cache append + causal attention (GQA), split between four paths
시그니처
- 입력: Q [..., Hq, Tq, headDim], K / V [..., Hkv, Tkv, headDim]
- 출력: out0 attention [..., Tq, Hq, headDim]
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
K |
input | |
Q |
input | |
V |
input | |
out0 |
output | |
K_cache |
persistent | |
V_cache |
persistent | |
partialML |
temporary | |
partialO |
temporary |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
headDim |
uint32_t |
— | |
numQHeads |
uint32_t |
— | |
numKVHeads |
uint32_t |
0 |
for GQA (Hkv < numQHeads); 0 → Hkv = numQHeads |
kvCacheSize |
uint32_t |
0 |
tokens the cache holds; 0 = no cache at all |
kvCacheDtype |
vai::Dtype |
vai::Dtype::None |
K / V cache; required when kvCacheSize > 0 |
kvCacheRing |
bool |
false |
the cache keeps the last kvCacheSize positions, rounded up to kvRingAlign |
동작 및 제약
Leading axes are more sequences swept in one dispatch; they ride the grid, so every count below stays per-sequence.
Q, K, V arrive as three separate buffers (so callers can transform Q/K but not V in between — e.g. RoPE). Supports GQA (numKVHeads < numQHeads). Q and the K / V inputs may differ in dtype (typeI / typeKV): a net may feed K / V in the cache dtype (llama3 / gemma3 do). On Intel K / V must come in Q's dtype (asserted).
- coopmat FA (attention-fa-cm): subgroup-MMA flash attention, fixed 16x64 tile; prefill and GQA-packed decode on cm-capable devices, and the only FA tile past Dh 256 (384 / 512 on NVIDIA / AMD).
- scalar FA (attention-fa): flash attention with per-(shape, vendor) tuned tiles, Dh <= 256; everything the cm gates reject.
- Intel FA (attention-fa-intel): Intel parts without a coopmat path; a 16-row tile packs one KV group's heads, subgroup-broadcast products.
- naive (attention): the unconditional fallback — unfamiliar devices degrade instead of crashing. See supportsFlashAttention().
Intel's coopmat tiles (Xe2 / Xe3) run attention-fa-cm-intel, the kernel Intel was verified with, under the tuning it was verified with; the other vendors' tuning moved attention-fa-cm on.
The path and tuning are selected per (shape, device) in prepare(); a tuning change re-requests the pipeline from the pool. Every path leaves the KV cache in the identical state (fused append), so the graph meaning does not depend on which one runs.
The mask's span bounds a query to its last span keys, a sliding window (e.g. Gemma3's local blocks). A ring cache (kvCacheRing) then keeps only the last kvCacheSize positions, position p at row p % kvCacheSize; the counter keeps counting positions. A dispatch overwrites the rows of its Tkv oldest positions, so kvCacheSize >= span + Tkv - 1 keeps every query's span whole. The node rounds a ring up to kvRingAlign, so no key tile wraps it.
kvCacheSize = 0 is orthogonal to both: it drops the cache itself, so a dispatch attends its own tokens and nothing else (an encoder, e.g. SigLIP). Nothing is allocated and no counter is wired — the cache and counter descriptors go null, and the 0 a null counter reads back is the whole trick.
Tkv may differ from Tq: queries and keys align at their ends, so query t reaches the keys up to its own mask end shifted by Tkv - Tq (e.g. a decoder over its encoder's output, Tq queries on Tkv keys). With a cache, K / V may stay unwired (Tkv = 0): the dispatch attends the cache alone and appends nothing, and the model keeps the counter at the rows held (prepare() asserts it is not 0, except on Intel: an empty cache leaves no key).
No mask (setCausalMask never called) binds the mask null: every query attends every key it reaches.
Split-K: when the grid would underfill the GPU, the keys a tile reaches (the cache and the new rows) are cut into kCount spans accumulating partials (partialO/partialML) that a second pass (attention-fa-reduce; attention-fa-reduce-intel on Intel or without subgroup arithmetic) folds with the online-softmax merge. kCount is the dispatch grid, so it holds for the whole recording: the vendor's tuner takes it from the grid's occupancy, the cache capacity and the keys prepare() sees. Where the spans fall follows the cache, which grows under a recording, so the kernel derives that from kvCacheCount per forward. The coopmat kernel's appendBeforeSweep cuts the cache alone and leaves the new token to the last split.
커널
vai/shaders/attention-fa-intel.compvai/shaders/attention-fa-cm-intel.compvai/shaders/attention-fa-cm.compvai/shaders/attention-fa.compvai/shaders/attention-fa-reduce-intel.compvai/shaders/attention-fa-reduce.compvai/shaders/attention.comp
Concat¶
join inputs along one axis
시그니처
- 입력: in0 .. inN [same shape except along the concat axis]
- 출력: out0 the inputs joined along axis, in input order
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0, in1, … |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
axis |
int |
0 |
negative counts from the end |
inputCount |
uint32_t |
2 |
|
allowPartialInputs |
bool |
false |
동작 및 제약
All inputs must have identical dtypes, ranks, and dimensions except along axis. A negative axis counts from the end.
What the axes ahead of the cut multiply to decides the cost, not the axis number. At 1 each input is one contiguous run of the output, so buffer copies place them and no shader runs — axis 0 is always this case, and so is a deeper axis with nothing but size-1 axes in front. One input arriving is then the whole output and out0 aliases it. Past 1 the runs interleave and a gather kernel reads them, which is where the 8-input limit and the dtype conversion live.
allowPartialInputs lets a recording bring only some of them; the rest are not joined. Off, every input is required, so a recording that cannot supply one drops this node.
커널
vai/shaders/concat.comp
Convolution¶
2D convolution with device-selected direct, implicit, or Winograd plans
시그니처
- 입력: in0 [..., Ci, Hi, Wi], optional residual [..., Co, Ho, Wo]
- 파라미터: weight [Co, Ci, K, K], optional scale [Co] and bias [Co]
- 출력: out0 [..., Co, ceil(Hi / stride), ceil(Wi / stride)]
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
residual |
input | |
out0 |
output | |
bias |
persistent | |
scale |
persistent | |
weight |
persistent | |
split |
temporary | |
winM |
temporary | |
winV |
temporary |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
inChannels |
uint32_t |
— | |
outChannels |
uint32_t |
— | |
kernelSize |
uint32_t |
— | |
stride |
uint32_t |
— | |
acceptResidual |
bool |
false |
input "residual" |
residualOrder |
vai::ResidualOrder |
vai::ResidualOrder::AfterEpilogue |
|
epilogue |
vai::Epilogue |
vai::Epilogue::None |
동작 및 제약
The "scale" and "bias" slots always exist; each applies only when the weight loader filled it. Odd kernels use symmetric same-padding, so each spatial output size is ceil(inputSize / stride). acceptResidual opens the "residual" input (input dtype), added after the epilogue or, with ResidualOrder::BeforeEpilogue, before it. The axes ahead of Ci are a batch of images. The plan is one image's; every dispatch stacks the images on its z axis, and each kernel offsets its buffers to its image.
커널
vai/shaders/conv-k1s1-gemm-cm.compvai/shaders/conv-k1s1-gemm.compvai/shaders/conv-k3s2-gemm-cm.compvai/shaders/conv-k3s1-implicit-gemm-cm.compvai/shaders/conv-k3s1-implicit-gemm.compvai/shaders/conv-implicit-gemm.compvai/shaders/conv-implicit-gemm-cm.compvai/shaders/conv-k3s2-gemm.compvai/shaders/cm-transform.compvai/shaders/conv-k3s1-winograd-filter-transform.compvai/shaders/conv-k3s1-winograd-input.compvai/shaders/conv-k3s1-winograd-output.compvai/shaders/bn-reduce.compvai/shaders/conv-k3s1-winograd-gemm.compvai/shaders/conv-k3s1-winograd-gemm-cm.comp
CyclicTableReader¶
read successive table rows, wrapping at the end
소스: vai/nodes/cyclic-table-reader.h
시그니처
- 입력: in0 [*] only scheduling: the node runs when it is fed
- 파라미터: table [numRows, ...] rows read in cyclic order, at the loader's dtype
- 출력: out0 table's shape without the leading axis, at oDtype
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output | |
table |
persistent |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
numRows |
uint32_t |
— | number of rows in the table |
동작 및 제약
Run k after rewind() copies row k % numRows, so a loop whose per-step values depend on the step alone (e.g. pi0.5's adaRMS modulation) reads them with no host write. rewind() starts over.
커널
vai/shaders/cyclic-table-reader.comp
EleAffine¶
element-wise affine, O = scale * in0 + shift
시그니처
- 입력: in0 [..., D] scale broadcast to in0's shape (right-aligned, size-1 axes broadcast); scale 1 when absent shift likewise; shift 0 when absent
- 출력: out0 in0's shape
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
scaleAsInput |
bool |
false |
input instead of persistent |
shiftAsInput |
bool |
false |
동작 및 제약
scale / shift are inputs under scaleAsInput / shiftAsInput, otherwise parameters at the loader's dtype. Each applies when present: filled by the loader, or wired; at least one has to be. Each may broadcast over any axes as long as the axes it keeps are adjacent, e.g. [1, D] over [T, D], or [C, 1, 1] over [C, H, W].
An add is shift alone, a multiply scale alone, and a gated residual x + g * y both, EleAffine(in0 = y, scale = g, shift = x) (e.g. pi0.5's adaRMS blocks).
커널
vai/shaders/ele-affine.comp
Flatten¶
Flatten tensor to 1D
시그니처
- 입력: in0 [any shape]
- 출력: out0 [N] where N = total number of elements (shares buffer with input)
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
동작 및 제약
elements is the input's total element count. The output aliases the input buffer.
Gate¶
conditional branch (no computation)
소스: vai/nodes/gate.h
시그니처
- 입력: in0 [any shape]
- 출력: then [same shape], else [same shape] — both alias in0's buffer
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
else |
output | |
then |
output |
동작 및 제약
One branch is recorded per command buffer. The engine drops every node that only the closed branch reaches, so those nodes neither prepare() nor run(); a node both branches reach (the merge point and everything past it) is kept. Leaving "else" unwired makes a closed gate drop the whole downstream.
The condition is host state, so flipping it re-records. Weights are uploaded by uploadModelParams(), which ignores gates — a branch closed on the first forward still gets its weights on the device.
Two producers may share one input slot only where a gate's branches merge; sortNodes() rejects any other fan-in. The branches need not agree on shape or dtype: a flip re-records, so every recording derives the merge point and its downstream from whichever branch survived it.
GatedMul¶
element-wise multiply with one operand activated (GLU gating)
시그니처
- 입력: in0 [..., 2H]
- 출력: out0 [..., H] = activation(in0[..., j]) * in0[..., H + j]
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
activation |
vai::Epilogue |
vai::Epilogue::Silu |
동작 및 제약
The two halves arrive in ONE buffer: the projections feeding them are a single Linear over the concatenated weights, so the leading half is whatever that concatenation put first. It is the activated one.
activation names the GLU variant — silu: SwiGLU, gelu: GEGLU, relu: ReGLU, none: bilinear.
커널
vai/shaders/gated-mul.comp
GroupNorm¶
group normalization over a CHW feature map
시그니처
- 입력: in0 [..., C, H, W], optional residual [..., C, H, W]
- 출력: epilogue(scale[c] * (in0 - μ[g(c)]) * rsqrt(σ²[g(c)] + eps) + shift[c]) + residual
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
residual |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
groups |
uint32_t |
— | |
eps |
float |
1e-5f |
|
coeffsAsInputs |
bool |
false |
scale / shift are inputs, not parameters |
acceptResidual |
bool |
false |
input "residual" |
residualOrder |
vai::ResidualOrder |
vai::ResidualOrder::AfterEpilogue |
|
epilogue |
vai::Epilogue |
vai::Epilogue::None |
동작 및 제약
The channel axis is third from the end; the axes before it are independent
samples. scale / shift [C] are parameters, or inputs under coeffsAsInputs
(e.g. AdaGN's timestep modulation); shift applies when present, filled by
the loader or wired. acceptResidual opens the "residual" input (input
dtype), added after the epilogue or, with ResidualOrder::BeforeEpilogue,
before it. The channels split into groups consecutive groups; each group's statistics
are computed at run time over its C/groups channels and all of their spatial
positions (unlike BatchNorm, which reads μ/σ² trained offline). One
workgroup per group (group-norm), statistics from a shifted 1-pass
reduction accumulated in fp32 regardless of I/O dtypes — the reduction is
subgroupAdd two-level when the device has subgroup arithmetic, a
shared-memory tree otherwise.
Both the workgroup size and the shape itself are baked into the pipeline, so it is requested in prepare() once the shape is known and re-requested when any of that changes (a no-op while it stays put) — same arrangement as LayerNorm. The shape is in there because the kernel's per-element channel index is j / HW: a push-constant HW costs an integer division per element, a specialization-constant one is a shift or a multiply-high. That makes the pipeline key per (dtype, group geometry), which is a handful of entries for a model — a ResNet-18 stage or a UNet1D level each contribute one, and every block inside it reuses that.
커널
vai/shaders/group-norm.comp
Identity¶
Pass-through node (no computation)
시그니처
- 입력: in0 [any shape]
- 출력: out0 [same shape] (shares buffer with input)
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
동작 및 제약
The output aliases the input buffer; no computation or copy occurs.
IndexCopy¶
write source slices at listed indices, as torch.index_copy
시그니처
- 입력: in0 [d0, .., d(r-1)] in1 in0's shape with d(axis) replaced by N
- 출력: out0 in0 with out0[.., index[n], ..] = in1[.., n, ..] along axis
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
in1 |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
axis |
uint32_t |
0 |
동작 및 제약
The index is N uint32 in a host-visible buffer the model owns and hands over with setIndex. It is read at dispatch time, so rewriting it after sync() needs no re-recording. An index past d(axis) leaves its slice unwritten.
out0 aliases in0 and only the N slices are written (= index_copy_), so no other node may read in0 after this one. in1 may be left out of a recording, and out0 then aliases in0 with nothing recorded. Bytes move as they are, so in0 and in1 share a dtype.
커널
vai/shaders/index-copy.comp
LayerNorm¶
layer normalization over the feature axis, single kernel
시그니처
- 입력: activations [..., D]
- 출력: normalized [..., D]
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
residual |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
eps |
float |
1e-5f |
|
coeffsAsInputs |
bool |
false |
scale / shift are inputs, not parameters |
acceptResidual |
bool |
false |
input "residual", added to the output |
동작 및 제약
scale / shift [D] (or [1, D] as an input); parameters, or inputs computed upstream under Config::coeffsAsInputs (e.g. DiT's adaLN). shift applies when present: filled by the loader, or wired.
O[i, j] = (I[i, j] - mean(I[i, :])) * rsqrt(var(I[i, :]) + eps) * scale[j] + shift[j]
One workgroup per row (layer-norm); the row statistics come from a shifted 1-pass reduction (Chan–Golub–LeVeque) accumulated in fp32 regardless of I/O dtypes. Two spec-selected variants share the kernel:
- reduction: subgroupAdd two-level when the device has subgroup arithmetic (subgroup size pinned to max — Intel varies it otherwise), shared-memory tree fallback everywhere else.
- loads: vec4 when I and O are fp32 and featSize % 4 == 0, scalar otherwise.
Geometry / reduction / vec4 are selected per (featSize, device) in lnTuning(); the workgroup size depends on featSize, so the pipeline is requested in prepare() once the shape is known, and a tuning change re-requests it from the pool (no-op while the tuning stays put).
커널
vai/shaders/layer-norm.comp
Linear¶
O = I · Wᵀ, split between two output decompositions
시그니처
- 입력: [..., inDim]
- 출력: [..., outDim]
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
residual |
input | 옵션에 따라 존재 |
out0 |
output | |
bias |
persistent | |
weight |
persistent | |
partial |
temporary |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
inDim |
uint32_t |
— | |
outDim |
uint32_t |
— | |
acceptResidual |
bool |
false |
input "residual" |
residualOrder |
vai::ResidualOrder |
vai::ResidualOrder::AfterEpilogue |
|
epilogue |
vai::Epilogue |
vai::Epilogue::None |
|
scalarScale |
float |
1.0f |
s, right after the epilogue |
splitSize |
uint32_t |
0 |
split-K: K length per split (0: the vendor's count per shape) |
동작 및 제약
vai naming: O [M, N], I [M, K] (M = flattened batch), W [N, K] (N = outDim, K = inDim).
- slab (M <= 8, linear-slab): a workgroup owns wgN whole columns of O, so K is split across its threads and reduced; bias / epilogue are fused in-kernel. See useSlabPath().
- tile (M > 8): a workgroup owns a wgM x wgN tile. Two kernels share the frame, chosen per device in selectLinearTile(): scalar (linear-tile, each thread accumulates its own outputs) and cooperative-matrix (linear-cm, subgroup MMA tiles; NVIDIA's Fill tiles run linear-cm-nv, which masks the K tail per k-block).
- AMD on RADV : linear-cm-amd for the tuned shapes, as a wave32 lane schedule (amdLinearLaneTile) or, on Strix, a cooperative schedule with its own load orders (amdLinearTile), reading W packed once on the GPU (linear-preshuffle, packWeightAmd). Other shapes keep linear-cm, on those chips also M-first or prefetching.
- Intel (useIntelTile; every M, slab included): linear-tile-intel, a port of OpenVINO's fully_connected_gpu_bf_tiled, reading W packed once on the GPU (packWeightIntel); its tile choice is intelTileParams(), its split intelTileSplitCount(). Everything below applies to every tile kernel:
The tile params are selected per output shape in prepare(); when a shape change selects a different tile, the pipeline is re-requested from the pool.
Split-K (splitSize option; 0 selects the vendor's rule per shape) accumulates partials P[splitCount, M, N] (oDtype) in a temporary buffer and folds them in a second pass (linear-reduce); bias / epilogue / the residual live in that reduce pass, unless linear-cm-amd fuses them into an unsplit tile.
O = s · epilogue(I · Wᵀ + bias) + R residualOrder AfterEpilogue O = s · epilogue(I · Wᵀ + bias + R) residualOrder BeforeEpilogue
s is Config::scalarScale (e.g. dt of an Euler step x + dt · v), 1 by default.
"bias" (its own dtype) is a parameter and "residual" R (oDtype) an input. Each is [M, N] or rows that repeat over M: [N], or one image's position table under a batch. The "bias" slot always exists; the bias applies only when the weight loader filled it.
커널
vai/shaders/linear-pack-intel.compvai/shaders/linear-preshuffle.compvai/shaders/linear-slab.compvai/shaders/linear-tile-intel.compvai/shaders/linear-cm-amd.compvai/shaders/linear-cm-nv.compvai/shaders/linear-cm.compvai/shaders/linear-tile.compvai/shaders/linear-reduce.comp
Mul¶
Element-wise multiplication
소스: vai/nodes/mul.h
시그니처
- 입력: in0 [any shape], in1 [same shape]
- 출력: out0 [same shape] = in0 * in1
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
in1 |
input | |
out0 |
output |
동작 및 제약
Inputs must have identical shapes and dtypes.
커널
vai/shaders/mul.comp
NearestUpsample¶
Replicates CHW pixels by an integer scale factor.
소스: vai/nodes/nearest-upsample.h
시그니처
- 입력: in0 [C, H, W]
- 출력: out0 [C, Hscale, Wscale]
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
동작 및 제약
scaleFactor must be greater than one. Each input pixel is replicated without interpolation.
커널
vai/shaders/upsample.comp
Permute¶
Permutes the axes of a tensor of any rank.
시그니처
- 입력: in0 [D0, .., Dr-1], r = perms.size()
- 출력: axes reordered by perms; reshape-only fast path when layout is unchanged
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
동작 및 제약
Output dimension i is input dimension perms[i], and perms must be a permutation of {0, .., r-1} with r at most PERMUTE_MAX_RANK. When the permutation preserves memory order and the dtype is unchanged, the output aliases the input; otherwise it uses new contiguous storage.
The rank is a specialization constant, so the pool holds one pipeline per rank.
커널
vai/shaders/permute.comp
Pool2d¶
2D pooling with a square kernel (MaxPool2d, AvgPool2d)
시그니처
- 입력: in0 [..., C, H, W], or [..., H, W, C] with channelsLast
- 출력: [..., C, Ho, Wo], or [..., Ho, Wo, C] where Ho = (H + 2*padding - kernelSize) / stride + 1, rounded up with ceilMode
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
kernelSize |
uint32_t |
— | |
stride |
uint32_t |
— | |
padding |
uint32_t |
0 |
|
ceilMode |
bool |
false |
a last window that would start in the right padding is dropped |
countIncludePad |
bool |
true |
Avg: padding counts in the divisor, as PyTorch's default |
channelsLast |
bool |
false |
동작 및 제약
channelsLast reads a token grid as it is laid out (rows of embedDim, e.g. a ViT's patches viewed as [G, G, D]), so no transpose is needed around the pool.
커널
vai/shaders/pool2d.comp
Reshape¶
Reshape tensor to specified dimensions
시그니처
- 입력: in0 [any shape]
- 출력: out0 [dims] (shares buffer with input)
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
동작 및 제약
A dim is a size, 0 for the input's dimension at that same position, or -1 for whatever the element count leaves over, at most once (= ONNX). 0 carries a size only the caller's data knows without the graph naming it. The resolved dims must multiply to the input's element count. The output aliases the input buffer.
RmsNorm¶
RMS normalization over the feature axis, single kernel
시그니처
- 입력: in0 activations [..., D]
- 출력: out0 normalized [..., D]
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
residual |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
eps |
float |
1e-5f |
|
coeffsAsInputs |
bool |
false |
scale / shift are inputs, not parameters |
acceptResidual |
bool |
false |
input "residual", added to the output |
동작 및 제약
scale / shift [D] (or [1, D] as an input), each applied when present: a parameter the loader filled, or an input wired upstream under Config::coeffsAsInputs (e.g. pi0.5's adaRMS) residual [..., D] as in0, an input under Config::acceptResidual
O[i, j] = I[i, j] * rsqrt(mean(I[i, :]^2) + eps) (* scale[j]) (+ shift[j]) (+ residual[i, j])
The row statistic is a 1-pass sum of squares accumulated in fp32 regardless of I/O dtypes. The workgroup size depends on D, so the pipeline is requested in prepare() once the shape is known.
커널
vai/shaders/rms-norm.comp
Rope¶
rotary position embedding (half-split rotation)
소스: vai/nodes/rope.h
시그니처
- 입력: in0 [.., H, Tq, Dh] (Q or K, Dh even)
- 파라미터: freqs [Dh/2] per-dimension angular frequency, built by the constructor
- 출력: out0 [.., H, Tq, Dh]
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output | |
freqs |
persistent |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
cfg |
RopeConfig |
— | frequencies; describe the whole model |
headDim |
uint32_t |
— | Dh, must be even |
동작 및 제약
The rotation depends on the token position alone, so anything ahead of the head axis is one more sweep and folds into the head count.
Positions continue from setKvCacheCounter()'s buffer, so a KV-cache decode step rotates by the absolute position rather than by its offset in this forward.
cfg.type picks how the frequencies are scaled: Default divides every one by factor (1 leaves them alone), and Llama3 divides only the long wavelengths, interpolating the band between the two frequency bounds.
커널
vai/shaders/rope.comp
RowSliceScatter¶
write slices into rows, as torch.slice_scatter on dim 0
소스: vai/nodes/row-slice-scatter.h
시그니처
- 입력: in0 [T, D] in1 [B, S, D] or [S, D] (S == sliceSize; B slices, one when rank 2)
- 출력: out0 [T, D], in0 with rows [rowStarts[b], rowStarts[b] + S) taken from slice b
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
in1 |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
sliceSize |
uint32_t |
— | S: rows in one slice; 0 when in1 is never fed |
동작 및 제약
in1 may be left out of a recording, and out0 then aliases in0 with nothing recorded. Rows move as bytes, so in0 and in1 share a dtype. The row starts are host state that the recorded copies bake in: setRowStarts re-records only when they change.
Slice¶
Take a run of consecutive indices along one axis
시그니처
- 입력: in0 [d0, .., d(r-1)] (rank >= 2)
- 출력: out0 the same shape with d(axis) replaced by count
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
axis |
uint32_t |
0 |
the axis to cut; 0 takes a contiguous block |
start |
int64_t |
— | first index on the axis; negative counts from the end |
count |
uint32_t |
1 |
how many consecutive indices to take |
동작 및 제약
A negative start counts from the end, with -1 selecting the last index.
What the axes ahead of the cut multiply to decides the cost, not the axis number. At 1 the run is one contiguous block, so out0 aliases it when the offset satisfies device alignment and is one buffer copy otherwise — axis 0 is always this case, and so is a deeper axis with nothing but size-1 axes in front. Past 1 the run is strided, count indices every d(axis), and a gather kernel reads it out. The shape decides the path; the caller writes the same node either way.
커널
vai/shaders/slice.comp
TokenEmbedding¶
token embedding, optionally fused with positional embedding
소스: vai/nodes/token-embedding.h
시그니처
- 입력: in0 token IDs [..., T] (uint32)
- 출력: out0 embedding [..., T, D] (fp32)
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output | |
posWeight |
persistent | |
weight |
persistent |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
vocabSize |
uint32_t |
— | |
embedDim |
uint32_t |
— | |
posTableSize |
uint32_t |
0 |
learned positional table rows; 0 = token only |
scale |
float |
1.0f |
multiplies the looked-up embedding |
동작 및 제약
Leading axes are more sequences swept in one dispatch, the way a Linear takes its leading axes as more rows, so the output keeps the input's rank.
posTableSize == 0 : O[n] = scale * W[ids[n]] (e.g. Llama3 — RoPE) posTableSize > 0 : O[n] = scale * W[ids[n]] + posW[cachedCount + n] (e.g. GPT-2)
Only Gemma uses scale (sqrt(D)). It stays out of the weights because a tied LM head reads the same table. posTableSize > 0 adds a "posWeight" [posTableSize, D] slot and needs setKvCacheCounter() to supply cachedCount; positions restart at each sequence, which the kernel only has to work out when there is more than one.
커널
vai/shaders/token-embedding.comp
TopKSampling¶
row-wise top-k sampling, one kernel run as two passes
시그니처
- 입력: logits [.., N]
- 출력: out0 [..] (one raw uint32 sampled index per row, in 4-byte slots)
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output | |
cands |
temporary |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
k |
uint32_t |
40 |
|
temperature |
float |
1.0f |
동작 및 제약
Every axis ahead of N is a row, the way a Linear takes its leading axes, and they come back on the output where they went in.
Keeps the k largest logits of each row, applies a temperature softmax over them, and samples by inverse CDF with a host-provided uniform refreshed every forward (setUniform()).
Two dispatches of one kernel: G workgroups each select the top-k of one ceil(N/G)-logit slice — exact while the slice stays register-resident — then one workgroup merges the G*k candidates and samples. Small rows (G == 1) skip the second dispatch.
Geometry (workgroup size, register list length), the reduction variant and G are selected per (shape, k, device) in topkTuning(); the workgroup size is a spec constant, so the pipeline is requested in prepare() once the shape is known, and a tuning change re-requests it from the pool (no-op while the tuning stays put).
커널
vai/shaders/topk-sampling.comp
TypeCast¶
Casts each tensor element to another dtype.
시그니처
- 입력: in0 [any shape], any dtype
- 출력: out0 [same shape] (oDtype)
슬롯
| 슬롯 | 종류 | 비고 |
|---|---|---|
in0 |
input | |
out0 |
output |
Config
| 필드 | 타입 | 기본값 | 설명 |
|---|---|---|---|
oDtype |
vai::Dtype |
— | required: the dtype to cast to |
동작 및 제약
in0 already at oDtype makes out0 the input tensor itself: no pipeline, no dispatch.
커널
vai/shaders/cast.comp