Skip to content

Built-in nodes

28 built-in nodes. See Node for the concept and Writing custom nodes and kernels for instructions on creating new nodes.

List

Node Summary Kernel
Activation Applies an element-wise activation. activation.comp
Add element-wise addition for residual connections add.comp
Argmax Row-wise argmax / greedy sampling argmax.comp
Attention fused KV cache append + causal attention (GQA), split between four paths attention-fa-intel.comp, attention-fa-cm-intel.comp, attention-fa-cm.comp, attention-fa.comp, attention-fa-reduce-intel.comp, attention-fa-reduce.comp, attention.comp
Concat join inputs along one axis concat.comp
Convolution 2D convolution with device-selected direct, implicit, or Winograd plans conv-k1s1-gemm-cm.comp, conv-k1s1-gemm.comp, conv-k3s2-gemm-cm.comp, conv-k3s1-implicit-gemm-cm.comp, conv-k3s1-implicit-gemm.comp, conv-implicit-gemm.comp, conv-implicit-gemm-cm.comp, conv-k3s2-gemm.comp, cm-transform.comp, conv-k3s1-winograd-filter-transform.comp, conv-k3s1-winograd-input.comp, conv-k3s1-winograd-output.comp, bn-reduce.comp, conv-k3s1-winograd-gemm.comp, conv-k3s1-winograd-gemm-cm.comp
CyclicTableReader read successive table rows, wrapping at the end cyclic-table-reader.comp
EleAffine element-wise affine, O = scale * in0 + shift ele-affine.comp
Flatten Flatten tensor to 1D None
Gate conditional branch (no computation) None
GatedMul element-wise multiply with one operand activated (GLU gating) gated-mul.comp
GroupNorm group normalization over a CHW feature map group-norm.comp
Identity Pass-through node (no computation) None
IndexCopy write source slices at listed indices, as torch.index_copy index-copy.comp
LayerNorm layer normalization over the feature axis, single kernel layer-norm.comp
Linear O = I · Wᵀ, split between two output decompositions linear-pack-intel.comp, linear-preshuffle.comp, linear-slab.comp, linear-tile-intel.comp, linear-cm-amd.comp, linear-cm-nv.comp, linear-cm.comp, linear-tile.comp, linear-reduce.comp
Mul Element-wise multiplication mul.comp
NearestUpsample Replicates CHW pixels by an integer scale factor. upsample.comp
Permute Permutes the axes of a tensor of any rank. permute.comp
Pool2d 2D pooling with a square kernel (MaxPool2d, AvgPool2d) pool2d.comp
Reshape Reshape tensor to specified dimensions None
RmsNorm RMS normalization over the feature axis, single kernel rms-norm.comp
Rope rotary position embedding (half-split rotation) rope.comp
RowSliceScatter write slices into rows, as torch.slice_scatter on dim 0 None
Slice Take a run of consecutive indices along one axis slice.comp
TokenEmbedding token embedding, optionally fused with positional embedding token-embedding.comp
TopKSampling row-wise top-k sampling, one kernel run as two passes topk-sampling.comp
TypeCast Casts each tensor element to another dtype. cast.comp

Activation

Applies an element-wise activation.

Source: vai/nodes/activation.h

Signature

  • Input: in0 [any shape]
  • Output: epilogue(in0); optionally in-place

Slots

Slot Type Notes
in0 input
out0 output

Config

Field Type Default Description
epilogue vai::Epilogue vai::Epilogue::None
inPlace bool false

Behavior and constraints

The inPlace option aliases the output to the input and requires matching input and output dtypes. Otherwise, the output uses separate storage.

Kernels

  • vai/shaders/activation.comp

Add

element-wise addition for residual connections

Source: vai/nodes/add.h

Signature

  • Input: in0, in1 [same shape]
  • Output: in0 + in1

Slots

Slot Type Notes
in0 input
in1 input
out0 output

Behavior and constraints

Inputs must have identical shapes and dtypes.

Kernels

  • vai/shaders/add.comp

Argmax

Row-wise argmax / greedy sampling

Source: vai/nodes/argmax.h

Signature

  • Input: in0 [M, N] (fp32 logits)
  • Output: out0 [M] (one raw uint32 index per row, in 4-byte slots)

Slots

Slot Type Notes
in0 input
out0 output

Behavior and constraints

Each row is reduced independently. out0[m] is the index of the largest value in in0[m, :]. Output elements are raw uint32 values, not fp32 numeric values.

Kernels

  • vai/shaders/argmax.comp

Attention

fused KV cache append + causal attention (GQA), split between four paths

Source: vai/nodes/attention.h

Signature

  • Input: Q [..., Hq, Tq, headDim], K / V [..., Hkv, Tkv, headDim]
  • Output: out0 attention [..., Tq, Hq, headDim]

Slots

Slot Type Notes
K input
Q input
V input
out0 output
K_cache persistent
V_cache persistent
partialML temporary
partialO temporary

Config

Field Type Default Description
headDim uint32_t Not specified
numQHeads uint32_t Not specified
numKVHeads uint32_t 0 for GQA (Hkv < numQHeads); 0 → Hkv = numQHeads
kvCacheSize uint32_t 0 tokens the cache holds; 0 = no cache at all
kvCacheDtype vai::Dtype vai::Dtype::None K / V cache; required when kvCacheSize > 0
kvCacheRing bool false the cache keeps the last kvCacheSize positions, rounded up to kvRingAlign

Behavior and constraints

Leading axes are more sequences swept in one dispatch; they ride the grid, so every count below stays per-sequence.

Q, K, V arrive as three separate buffers (so callers can transform Q/K but not V in between — e.g. RoPE). Supports GQA (numKVHeads < numQHeads). Q and the K / V inputs may differ in dtype (typeI / typeKV): a net may feed K / V in the cache dtype (llama3 / gemma3 do). On Intel K / V must come in Q's dtype (asserted).

  • coopmat FA (attention-fa-cm): subgroup-MMA flash attention, fixed 16x64 tile; prefill and GQA-packed decode on cm-capable devices, and the only FA tile past Dh 256 (384 / 512 on NVIDIA / AMD).
  • scalar FA (attention-fa): flash attention with per-(shape, vendor) tuned tiles, Dh <= 256; everything the cm gates reject.
  • Intel FA (attention-fa-intel): Intel parts without a coopmat path; a 16-row tile packs one KV group's heads, subgroup-broadcast products.
  • naive (attention): the unconditional fallback — unfamiliar devices degrade instead of crashing. See supportsFlashAttention().

Intel's coopmat tiles (Xe2 / Xe3) run attention-fa-cm-intel, the kernel Intel was verified with, under the tuning it was verified with; the other vendors' tuning moved attention-fa-cm on.

The path and tuning are selected per (shape, device) in prepare(); a tuning change re-requests the pipeline from the pool. Every path leaves the KV cache in the identical state (fused append), so the graph meaning does not depend on which one runs.

The mask's span bounds a query to its last span keys, a sliding window (e.g. Gemma3's local blocks). A ring cache (kvCacheRing) then keeps only the last kvCacheSize positions, position p at row p % kvCacheSize; the counter keeps counting positions. A dispatch overwrites the rows of its Tkv oldest positions, so kvCacheSize >= span + Tkv - 1 keeps every query's span whole. The node rounds a ring up to kvRingAlign, so no key tile wraps it.

kvCacheSize = 0 is orthogonal to both: it drops the cache itself, so a dispatch attends its own tokens and nothing else (an encoder, e.g. SigLIP). Nothing is allocated and no counter is wired — the cache and counter descriptors go null, and the 0 a null counter reads back is the whole trick.

Tkv may differ from Tq: queries and keys align at their ends, so query t reaches the keys up to its own mask end shifted by Tkv - Tq (e.g. a decoder over its encoder's output, Tq queries on Tkv keys). With a cache, K / V may stay unwired (Tkv = 0): the dispatch attends the cache alone and appends nothing, and the model keeps the counter at the rows held (prepare() asserts it is not 0, except on Intel: an empty cache leaves no key).

No mask (setCausalMask never called) binds the mask null: every query attends every key it reaches.

Split-K: when the grid would underfill the GPU, the keys a tile reaches (the cache and the new rows) are cut into kCount spans accumulating partials (partialO/partialML) that a second pass (attention-fa-reduce; attention-fa-reduce-intel on Intel or without subgroup arithmetic) folds with the online-softmax merge. kCount is the dispatch grid, so it holds for the whole recording: the vendor's tuner takes it from the grid's occupancy, the cache capacity and the keys prepare() sees. Where the spans fall follows the cache, which grows under a recording, so the kernel derives that from kvCacheCount per forward. The coopmat kernel's appendBeforeSweep cuts the cache alone and leaves the new token to the last split.

Kernels

  • vai/shaders/attention-fa-intel.comp
  • vai/shaders/attention-fa-cm-intel.comp
  • vai/shaders/attention-fa-cm.comp
  • vai/shaders/attention-fa.comp
  • vai/shaders/attention-fa-reduce-intel.comp
  • vai/shaders/attention-fa-reduce.comp
  • vai/shaders/attention.comp

Concat

join inputs along one axis

Source: vai/nodes/concat.h

Signature

  • Input: in0 .. inN [same shape except along the concat axis]
  • Output: out0 the inputs joined along axis, in input order

Slots

Slot Type Notes
in0, in1, … input
out0 output

Config

Field Type Default Description
axis int 0 negative counts from the end
inputCount uint32_t 2
allowPartialInputs bool false

Behavior and constraints

All inputs must have identical dtypes, ranks, and dimensions except along axis. A negative axis counts from the end.

What the axes ahead of the cut multiply to decides the cost, not the axis number. At 1 each input is one contiguous run of the output, so buffer copies place them and no shader runs — axis 0 is always this case, and so is a deeper axis with nothing but size-1 axes in front. One input arriving is then the whole output and out0 aliases it. Past 1 the runs interleave and a gather kernel reads them, which is where the 8-input limit and the dtype conversion live.

allowPartialInputs lets a recording bring only some of them; the rest are not joined. Off, every input is required, so a recording that cannot supply one drops this node.

Kernels

  • vai/shaders/concat.comp

Convolution

2D convolution with device-selected direct, implicit, or Winograd plans

Source: vai/nodes/convolution.h

Signature

  • Input: in0 [..., Ci, Hi, Wi], optional residual [..., Co, Ho, Wo]
  • Params: weight [Co, Ci, K, K], optional scale [Co] and bias [Co]
  • Output: out0 [..., Co, ceil(Hi / stride), ceil(Wi / stride)]

Slots

Slot Type Notes
in0 input
residual input
out0 output
bias persistent
scale persistent
weight persistent
split temporary
winM temporary
winV temporary

Config

Field Type Default Description
inChannels uint32_t Not specified
outChannels uint32_t Not specified
kernelSize uint32_t Not specified
stride uint32_t Not specified
acceptResidual bool false input "residual"
residualOrder vai::ResidualOrder vai::ResidualOrder::AfterEpilogue
epilogue vai::Epilogue vai::Epilogue::None

Behavior and constraints

The "scale" and "bias" slots always exist; each applies only when the weight loader filled it. Odd kernels use symmetric same-padding, so each spatial output size is ceil(inputSize / stride). acceptResidual opens the "residual" input (input dtype), added after the epilogue or, with ResidualOrder::BeforeEpilogue, before it. The axes ahead of Ci are a batch of images. The plan is one image's; every dispatch stacks the images on its z axis, and each kernel offsets its buffers to its image.

Kernels

  • vai/shaders/conv-k1s1-gemm-cm.comp
  • vai/shaders/conv-k1s1-gemm.comp
  • vai/shaders/conv-k3s2-gemm-cm.comp
  • vai/shaders/conv-k3s1-implicit-gemm-cm.comp
  • vai/shaders/conv-k3s1-implicit-gemm.comp
  • vai/shaders/conv-implicit-gemm.comp
  • vai/shaders/conv-implicit-gemm-cm.comp
  • vai/shaders/conv-k3s2-gemm.comp
  • vai/shaders/cm-transform.comp
  • vai/shaders/conv-k3s1-winograd-filter-transform.comp
  • vai/shaders/conv-k3s1-winograd-input.comp
  • vai/shaders/conv-k3s1-winograd-output.comp
  • vai/shaders/bn-reduce.comp
  • vai/shaders/conv-k3s1-winograd-gemm.comp
  • vai/shaders/conv-k3s1-winograd-gemm-cm.comp

CyclicTableReader

read successive table rows, wrapping at the end

Source: vai/nodes/cyclic-table-reader.h

Signature

  • Input: in0 [*] only scheduling: the node runs when it is fed
  • Params: table [numRows, ...] rows read in cyclic order, at the loader's dtype
  • Output: out0 table's shape without the leading axis, at oDtype

Slots

Slot Type Notes
in0 input
out0 output
table persistent

Config

Field Type Default Description
numRows uint32_t Not specified number of rows in the table

Behavior and constraints

Run k after rewind() copies row k % numRows, so a loop whose per-step values depend on the step alone (e.g. pi0.5's adaRMS modulation) reads them with no host write. rewind() starts over.

Kernels

  • vai/shaders/cyclic-table-reader.comp

EleAffine

element-wise affine, O = scale * in0 + shift

Source: vai/nodes/ele-affine.h

Signature

  • Input: in0 [..., D] scale broadcast to in0's shape (right-aligned, size-1 axes broadcast); scale 1 when absent shift likewise; shift 0 when absent
  • Output: out0 in0's shape

Slots

Slot Type Notes
in0 input
out0 output

Config

Field Type Default Description
scaleAsInput bool false input instead of persistent
shiftAsInput bool false

Behavior and constraints

scale / shift are inputs under scaleAsInput / shiftAsInput, otherwise parameters at the loader's dtype. Each applies when present: filled by the loader, or wired; at least one has to be. Each may broadcast over any axes as long as the axes it keeps are adjacent, e.g. [1, D] over [T, D], or [C, 1, 1] over [C, H, W].

An add is shift alone, a multiply scale alone, and a gated residual x + g * y both, EleAffine(in0 = y, scale = g, shift = x) (e.g. pi0.5's adaRMS blocks).

Kernels

  • vai/shaders/ele-affine.comp

Flatten

Flatten tensor to 1D

Source: vai/nodes/flatten.h

Signature

  • Input: in0 [any shape]
  • Output: out0 [N] where N = total number of elements (shares buffer with input)

Slots

Slot Type Notes
in0 input
out0 output

Behavior and constraints

elements is the input's total element count. The output aliases the input buffer.


Gate

conditional branch (no computation)

Source: vai/nodes/gate.h

Signature

  • Input: in0 [any shape]
  • Output: then [same shape], else [same shape] — both alias in0's buffer

Slots

Slot Type Notes
in0 input
else output
then output

Behavior and constraints

One branch is recorded per command buffer. The engine drops every node that only the closed branch reaches, so those nodes neither prepare() nor run(); a node both branches reach (the merge point and everything past it) is kept. Leaving "else" unwired makes a closed gate drop the whole downstream.

The condition is host state, so flipping it re-records. Weights are uploaded by uploadModelParams(), which ignores gates — a branch closed on the first forward still gets its weights on the device.

Two producers may share one input slot only where a gate's branches merge; sortNodes() rejects any other fan-in. The branches need not agree on shape or dtype: a flip re-records, so every recording derives the merge point and its downstream from whichever branch survived it.


GatedMul

element-wise multiply with one operand activated (GLU gating)

Source: vai/nodes/gated-mul.h

Signature

  • Input: in0 [..., 2H]
  • Output: out0 [..., H] = activation(in0[..., j]) * in0[..., H + j]

Slots

Slot Type Notes
in0 input
out0 output

Config

Field Type Default Description
activation vai::Epilogue vai::Epilogue::Silu

Behavior and constraints

The two halves arrive in ONE buffer: the projections feeding them are a single Linear over the concatenated weights, so the leading half is whatever that concatenation put first. It is the activated one.

activation names the GLU variant — silu: SwiGLU, gelu: GEGLU, relu: ReGLU, none: bilinear.

Kernels

  • vai/shaders/gated-mul.comp

GroupNorm

group normalization over a CHW feature map

Source: vai/nodes/group-norm.h

Signature

  • Input: in0 [..., C, H, W], optional residual [..., C, H, W]
  • Output: epilogue(scale[c] * (in0 - μ[g(c)]) * rsqrt(σ²[g(c)] + eps) + shift[c]) + residual

Slots

Slot Type Notes
in0 input
residual input
out0 output

Config

Field Type Default Description
groups uint32_t Not specified
eps float 1e-5f
coeffsAsInputs bool false scale / shift are inputs, not parameters
acceptResidual bool false input "residual"
residualOrder vai::ResidualOrder vai::ResidualOrder::AfterEpilogue
epilogue vai::Epilogue vai::Epilogue::None

Behavior and constraints

The channel axis is third from the end; the axes before it are independent samples. scale / shift [C] are parameters, or inputs under coeffsAsInputs (e.g. AdaGN's timestep modulation); shift applies when present, filled by the loader or wired. acceptResidual opens the "residual" input (input dtype), added after the epilogue or, with ResidualOrder::BeforeEpilogue, before it. The channels split into groups consecutive groups; each group's statistics are computed at run time over its C/groups channels and all of their spatial positions (unlike BatchNorm, which reads μ/σ² trained offline). One workgroup per group (group-norm), statistics from a shifted 1-pass reduction accumulated in fp32 regardless of I/O dtypes — the reduction is subgroupAdd two-level when the device has subgroup arithmetic, a shared-memory tree otherwise.

Both the workgroup size and the shape itself are baked into the pipeline, so it is requested in prepare() once the shape is known and re-requested when any of that changes (a no-op while it stays put) — same arrangement as LayerNorm. The shape is in there because the kernel's per-element channel index is j / HW: a push-constant HW costs an integer division per element, a specialization-constant one is a shift or a multiply-high. That makes the pipeline key per (dtype, group geometry), which is a handful of entries for a model — a ResNet-18 stage or a UNet1D level each contribute one, and every block inside it reuses that.

Kernels

  • vai/shaders/group-norm.comp

Identity

Pass-through node (no computation)

Source: vai/nodes/identity.h

Signature

  • Input: in0 [any shape]
  • Output: out0 [same shape] (shares buffer with input)

Slots

Slot Type Notes
in0 input
out0 output

Behavior and constraints

The output aliases the input buffer; no computation or copy occurs.


IndexCopy

write source slices at listed indices, as torch.index_copy

Source: vai/nodes/index-copy.h

Signature

  • Input: in0 [d0, .., d(r-1)] in1 in0's shape with d(axis) replaced by N
  • Output: out0 in0 with out0[.., index[n], ..] = in1[.., n, ..] along axis

Slots

Slot Type Notes
in0 input
in1 input
out0 output

Config

Field Type Default Description
axis uint32_t 0

Behavior and constraints

The index is N uint32 in a host-visible buffer the model owns and hands over with setIndex. It is read at dispatch time, so rewriting it after sync() needs no re-recording. An index past d(axis) leaves its slice unwritten.

out0 aliases in0 and only the N slices are written (= index_copy_), so no other node may read in0 after this one. in1 may be left out of a recording, and out0 then aliases in0 with nothing recorded. Bytes move as they are, so in0 and in1 share a dtype.

Kernels

  • vai/shaders/index-copy.comp

LayerNorm

layer normalization over the feature axis, single kernel

Source: vai/nodes/layer-norm.h

Signature

  • Input: activations [..., D]
  • Output: normalized [..., D]

Slots

Slot Type Notes
in0 input
residual input
out0 output

Config

Field Type Default Description
eps float 1e-5f
coeffsAsInputs bool false scale / shift are inputs, not parameters
acceptResidual bool false input "residual", added to the output

Behavior and constraints

scale / shift [D] (or [1, D] as an input); parameters, or inputs computed upstream under Config::coeffsAsInputs (e.g. DiT's adaLN). shift applies when present: filled by the loader, or wired.

O[i, j] = (I[i, j] - mean(I[i, :])) * rsqrt(var(I[i, :]) + eps) * scale[j] + shift[j]

One workgroup per row (layer-norm); the row statistics come from a shifted 1-pass reduction (Chan–Golub–LeVeque) accumulated in fp32 regardless of I/O dtypes. Two spec-selected variants share the kernel:

  • reduction: subgroupAdd two-level when the device has subgroup arithmetic (subgroup size pinned to max — Intel varies it otherwise), shared-memory tree fallback everywhere else.
  • loads: vec4 when I and O are fp32 and featSize % 4 == 0, scalar otherwise.

Geometry / reduction / vec4 are selected per (featSize, device) in lnTuning(); the workgroup size depends on featSize, so the pipeline is requested in prepare() once the shape is known, and a tuning change re-requests it from the pool (no-op while the tuning stays put).

Kernels

  • vai/shaders/layer-norm.comp

Linear

O = I · Wᵀ, split between two output decompositions

Source: vai/nodes/linear.h

Signature

  • Input: [..., inDim]
  • Output: [..., outDim]

Slots

Slot Type Notes
in0 input
residual input Present depending on config
out0 output
bias persistent
weight persistent
partial temporary

Config

Field Type Default Description
inDim uint32_t Not specified
outDim uint32_t Not specified
acceptResidual bool false input "residual"
residualOrder vai::ResidualOrder vai::ResidualOrder::AfterEpilogue
epilogue vai::Epilogue vai::Epilogue::None
scalarScale float 1.0f s, right after the epilogue
splitSize uint32_t 0 split-K: K length per split (0: the vendor's count per shape)

Behavior and constraints

vai naming: O [M, N], I [M, K] (M = flattened batch), W [N, K] (N = outDim, K = inDim).

  • slab (M <= 8, linear-slab): a workgroup owns wgN whole columns of O, so K is split across its threads and reduced; bias / epilogue are fused in-kernel. See useSlabPath().
  • tile (M > 8): a workgroup owns a wgM x wgN tile. Two kernels share the frame, chosen per device in selectLinearTile(): scalar (linear-tile, each thread accumulates its own outputs) and cooperative-matrix (linear-cm, subgroup MMA tiles; NVIDIA's Fill tiles run linear-cm-nv, which masks the K tail per k-block).
  • AMD on RADV : linear-cm-amd for the tuned shapes, as a wave32 lane schedule (amdLinearLaneTile) or, on Strix, a cooperative schedule with its own load orders (amdLinearTile), reading W packed once on the GPU (linear-preshuffle, packWeightAmd). Other shapes keep linear-cm, on those chips also M-first or prefetching.
  • Intel (useIntelTile; every M, slab included): linear-tile-intel, a port of OpenVINO's fully_connected_gpu_bf_tiled, reading W packed once on the GPU (packWeightIntel); its tile choice is intelTileParams(), its split intelTileSplitCount(). Everything below applies to every tile kernel:

The tile params are selected per output shape in prepare(); when a shape change selects a different tile, the pipeline is re-requested from the pool.

Split-K (splitSize option; 0 selects the vendor's rule per shape) accumulates partials P[splitCount, M, N] (oDtype) in a temporary buffer and folds them in a second pass (linear-reduce); bias / epilogue / the residual live in that reduce pass, unless linear-cm-amd fuses them into an unsplit tile.

O = s · epilogue(I · Wᵀ + bias) + R residualOrder AfterEpilogue O = s · epilogue(I · Wᵀ + bias + R) residualOrder BeforeEpilogue

s is Config::scalarScale (e.g. dt of an Euler step x + dt · v), 1 by default.

"bias" (its own dtype) is a parameter and "residual" R (oDtype) an input. Each is [M, N] or rows that repeat over M: [N], or one image's position table under a batch. The "bias" slot always exists; the bias applies only when the weight loader filled it.

Kernels

  • vai/shaders/linear-pack-intel.comp
  • vai/shaders/linear-preshuffle.comp
  • vai/shaders/linear-slab.comp
  • vai/shaders/linear-tile-intel.comp
  • vai/shaders/linear-cm-amd.comp
  • vai/shaders/linear-cm-nv.comp
  • vai/shaders/linear-cm.comp
  • vai/shaders/linear-tile.comp
  • vai/shaders/linear-reduce.comp

Mul

Element-wise multiplication

Source: vai/nodes/mul.h

Signature

  • Input: in0 [any shape], in1 [same shape]
  • Output: out0 [same shape] = in0 * in1

Slots

Slot Type Notes
in0 input
in1 input
out0 output

Behavior and constraints

Inputs must have identical shapes and dtypes.

Kernels

  • vai/shaders/mul.comp

NearestUpsample

Replicates CHW pixels by an integer scale factor.

Source: vai/nodes/nearest-upsample.h

Signature

  • Input: in0 [C, H, W]
  • Output: out0 [C, Hscale, Wscale]

Slots

Slot Type Notes
in0 input
out0 output

Behavior and constraints

scaleFactor must be greater than one. Each input pixel is replicated without interpolation.

Kernels

  • vai/shaders/upsample.comp

Permute

Permutes the axes of a tensor of any rank.

Source: vai/nodes/permute.h

Signature

  • Input: in0 [D0, .., Dr-1], r = perms.size()
  • Output: axes reordered by perms; reshape-only fast path when layout is unchanged

Slots

Slot Type Notes
in0 input
out0 output

Behavior and constraints

Output dimension i is input dimension perms[i], and perms must be a permutation of {0, .., r-1} with r at most PERMUTE_MAX_RANK. When the permutation preserves memory order and the dtype is unchanged, the output aliases the input; otherwise it uses new contiguous storage.

The rank is a specialization constant, so the pool holds one pipeline per rank.

Kernels

  • vai/shaders/permute.comp

Pool2d

2D pooling with a square kernel (MaxPool2d, AvgPool2d)

Source: vai/nodes/pool2d.h

Signature

  • Input: in0 [..., C, H, W], or [..., H, W, C] with channelsLast
  • Output: [..., C, Ho, Wo], or [..., Ho, Wo, C] where Ho = (H + 2*padding - kernelSize) / stride + 1, rounded up with ceilMode

Slots

Slot Type Notes
in0 input
out0 output

Config

Field Type Default Description
kernelSize uint32_t Not specified
stride uint32_t Not specified
padding uint32_t 0
ceilMode bool false a last window that would start in the right padding is dropped
countIncludePad bool true Avg: padding counts in the divisor, as PyTorch's default
channelsLast bool false

Behavior and constraints

channelsLast reads a token grid as it is laid out (rows of embedDim, e.g. a ViT's patches viewed as [G, G, D]), so no transpose is needed around the pool.

Kernels

  • vai/shaders/pool2d.comp

Reshape

Reshape tensor to specified dimensions

Source: vai/nodes/reshape.h

Signature

  • Input: in0 [any shape]
  • Output: out0 [dims] (shares buffer with input)

Slots

Slot Type Notes
in0 input
out0 output

Behavior and constraints

A dim is a size, 0 for the input's dimension at that same position, or -1 for whatever the element count leaves over, at most once (= ONNX). 0 carries a size only the caller's data knows without the graph naming it. The resolved dims must multiply to the input's element count. The output aliases the input buffer.


RmsNorm

RMS normalization over the feature axis, single kernel

Source: vai/nodes/rms-norm.h

Signature

  • Input: in0 activations [..., D]
  • Output: out0 normalized [..., D]

Slots

Slot Type Notes
in0 input
residual input
out0 output

Config

Field Type Default Description
eps float 1e-5f
coeffsAsInputs bool false scale / shift are inputs, not parameters
acceptResidual bool false input "residual", added to the output

Behavior and constraints

scale / shift [D] (or [1, D] as an input), each applied when present: a parameter the loader filled, or an input wired upstream under Config::coeffsAsInputs (e.g. pi0.5's adaRMS) residual [..., D] as in0, an input under Config::acceptResidual

O[i, j] = I[i, j] * rsqrt(mean(I[i, :]^2) + eps) (* scale[j]) (+ shift[j]) (+ residual[i, j])

The row statistic is a 1-pass sum of squares accumulated in fp32 regardless of I/O dtypes. The workgroup size depends on D, so the pipeline is requested in prepare() once the shape is known.

Kernels

  • vai/shaders/rms-norm.comp

Rope

rotary position embedding (half-split rotation)

Source: vai/nodes/rope.h

Signature

  • Input: in0 [.., H, Tq, Dh] (Q or K, Dh even)
  • Params: freqs [Dh/2] per-dimension angular frequency, built by the constructor
  • Output: out0 [.., H, Tq, Dh]

Slots

Slot Type Notes
in0 input
out0 output
freqs persistent

Config

Field Type Default Description
cfg RopeConfig Not specified frequencies; describe the whole model
headDim uint32_t Not specified Dh, must be even

Behavior and constraints

The rotation depends on the token position alone, so anything ahead of the head axis is one more sweep and folds into the head count.

Positions continue from setKvCacheCounter()'s buffer, so a KV-cache decode step rotates by the absolute position rather than by its offset in this forward.

cfg.type picks how the frequencies are scaled: Default divides every one by factor (1 leaves them alone), and Llama3 divides only the long wavelengths, interpolating the band between the two frequency bounds.

Kernels

  • vai/shaders/rope.comp

RowSliceScatter

write slices into rows, as torch.slice_scatter on dim 0

Source: vai/nodes/row-slice-scatter.h

Signature

  • Input: in0 [T, D] in1 [B, S, D] or [S, D] (S == sliceSize; B slices, one when rank 2)
  • Output: out0 [T, D], in0 with rows [rowStarts[b], rowStarts[b] + S) taken from slice b

Slots

Slot Type Notes
in0 input
in1 input
out0 output

Config

Field Type Default Description
sliceSize uint32_t Not specified S: rows in one slice; 0 when in1 is never fed

Behavior and constraints

in1 may be left out of a recording, and out0 then aliases in0 with nothing recorded. Rows move as bytes, so in0 and in1 share a dtype. The row starts are host state that the recorded copies bake in: setRowStarts re-records only when they change.


Slice

Take a run of consecutive indices along one axis

Source: vai/nodes/slice.h

Signature

  • Input: in0 [d0, .., d(r-1)] (rank >= 2)
  • Output: out0 the same shape with d(axis) replaced by count

Slots

Slot Type Notes
in0 input
out0 output

Config

Field Type Default Description
axis uint32_t 0 the axis to cut; 0 takes a contiguous block
start int64_t Not specified first index on the axis; negative counts from the end
count uint32_t 1 how many consecutive indices to take

Behavior and constraints

A negative start counts from the end, with -1 selecting the last index.

What the axes ahead of the cut multiply to decides the cost, not the axis number. At 1 the run is one contiguous block, so out0 aliases it when the offset satisfies device alignment and is one buffer copy otherwise — axis 0 is always this case, and so is a deeper axis with nothing but size-1 axes in front. Past 1 the run is strided, count indices every d(axis), and a gather kernel reads it out. The shape decides the path; the caller writes the same node either way.

Kernels

  • vai/shaders/slice.comp

TokenEmbedding

token embedding, optionally fused with positional embedding

Source: vai/nodes/token-embedding.h

Signature

  • Input: in0 token IDs [..., T] (uint32)
  • Output: out0 embedding [..., T, D] (fp32)

Slots

Slot Type Notes
in0 input
out0 output
posWeight persistent
weight persistent

Config

Field Type Default Description
vocabSize uint32_t Not specified
embedDim uint32_t Not specified
posTableSize uint32_t 0 learned positional table rows; 0 = token only
scale float 1.0f multiplies the looked-up embedding

Behavior and constraints

Leading axes are more sequences swept in one dispatch, the way a Linear takes its leading axes as more rows, so the output keeps the input's rank.

posTableSize == 0 : O[n] = scale * W[ids[n]] (e.g. Llama3 — RoPE) posTableSize > 0 : O[n] = scale * W[ids[n]] + posW[cachedCount + n] (e.g. GPT-2)

Only Gemma uses scale (sqrt(D)). It stays out of the weights because a tied LM head reads the same table. posTableSize > 0 adds a "posWeight" [posTableSize, D] slot and needs setKvCacheCounter() to supply cachedCount; positions restart at each sequence, which the kernel only has to work out when there is more than one.

Kernels

  • vai/shaders/token-embedding.comp

TopKSampling

row-wise top-k sampling, one kernel run as two passes

Source: vai/nodes/topk-sampling.h

Signature

  • Input: logits [.., N]
  • Output: out0 [..] (one raw uint32 sampled index per row, in 4-byte slots)

Slots

Slot Type Notes
in0 input
out0 output
cands temporary

Config

Field Type Default Description
k uint32_t 40
temperature float 1.0f

Behavior and constraints

Every axis ahead of N is a row, the way a Linear takes its leading axes, and they come back on the output where they went in.

Keeps the k largest logits of each row, applies a temperature softmax over them, and samples by inverse CDF with a host-provided uniform refreshed every forward (setUniform()).

Two dispatches of one kernel: G workgroups each select the top-k of one ceil(N/G)-logit slice — exact while the slice stays register-resident — then one workgroup merges the G*k candidates and samples. Small rows (G == 1) skip the second dispatch.

Geometry (workgroup size, register list length), the reduction variant and G are selected per (shape, k, device) in topkTuning(); the workgroup size is a spec constant, so the pipeline is requested in prepare() once the shape is known, and a tuning change re-requests it from the pool (no-op while the tuning stays put).

Kernels

  • vai/shaders/topk-sampling.comp

TypeCast

Casts each tensor element to another dtype.

Source: vai/nodes/type-cast.h

Signature

  • Input: in0 [any shape], any dtype
  • Output: out0 [same shape] (oDtype)

Slots

Slot Type Notes
in0 input
out0 output

Config

Field Type Default Description
oDtype vai::Dtype Not specified required: the dtype to cast to

Behavior and constraints

in0 already at oDtype makes out0 the input tensor itself: no pipeline, no dispatch.

Kernels

  • vai/shaders/cast.comp