Overview¶
vAi is a Vulkan inference engine built on a graph DSL. It assembles model graphs using a C++ DSL (a dedicated syntax for connecting operator nodes) and executes them using Vulkan Compute Shaders.
1. Features¶
1.1 Portability¶
A single codebase can run and be tested directly on multiple GPUs. Because vAi uses only Vulkan, the industry-standard GPU API, there is no need to rewrite code for vendor-specific SDKs from NVIDIA, AMD, Intel, Qualcomm, and others.
1.2 Lightweight¶
When deploying an LLM with vAi, the runtime binary is just over 2 MB. The dependency chain is designed for on-device environments. Selective builds include only the model code and operator kernels actually used, without additional configuration.
1.3 High performance¶
GPU command buffers are reused when conditions such as input shapes and buffer bindings remain the same. This requires very little CPU work during inference. Conventional inference engines typically traverse the computation graph and repeatedly submit kernel execution commands to the GPU on every inference, but vAi creates the command buffer once and reuses it. GPU kernel tuning also selects different kernels for different devices, so portability does not have to come at the cost of performance.
1.4 Intuitiveness¶
The graph DSL exposes the model structure directly in code. With vAi's dedicated graph syntax and object-oriented design, a model can be built by connecting Node and NodeGroup objects like LEGO bricks.
template<TransformerType layout, QkvLayout qkv>
struct TransformerBlock : public vai::NodeGroup
{
vai::node::Identity input;
typename BranchNorm<layout>::Type norm1;
MultiHeadAttention<qkv> mhAttn;
std::optional<vai::node::Slice> lastRow;
std::optional<vai::node::Gate> gate;
typename BranchNorm<layout>::Type norm2;
typename BranchFfn<layout>::Type ffn;
TransformerBlock(...)
{
input - norm1 - mhAttn - lastRow - gate/"then" - norm2 - ffn;
input - "residual"/mhAttn - lastRow - gate/"then" - "residual"/ffn;
}
};
The member list is the block's structure, and the connections are its data flow. The first line in the constructor draws the main path, and the second draws the residual path, like a design diagram. Because the code is the model's structural diagram, it is easy to read and modify.
1.5 Extensibility¶
New operators or layers with custom optimizations can be added directly. Inherit from Node and write the required kernel without modifying a single line of core code. Newly created kernels can be shared with anyone who needs them without waiting for a merge into the upstream project. There is no need to maintain a fork or conform to a plugin specification.
2. Project structure¶
vai/ Core engine (vAi library)
engine/
neural-net.h
neural-net.cpp Graph sorting, buffer preparation, command recording
graph-flow.h Connection DSL and NodeGroup
graph-controller.h Execution paths based on inputs and conditional branches
tensor.h
safetensors-parser.h
weights.h
buffer-pool.h
pipeline-pool.h
error.h
nodes/ Built-in nodes
groups/ Built-in groups
shaders/ Built-in kernels (.comp)
utils/
data/ Model checkpoints, input data, etc.
models/ Supported models
<impl>/<model-name>/
net.h Model definition
<distribution-name>-weights.h Checkpoint loader
demo.cpp demo-<model-name>
examples/ Programs beyond one model's demo (e.g., mnist/, chat-llama3/)
python/ Python bindings (pyvai)
benchmarks/
external/eva/ Vulkan wrapper (submodule)
docs/
Start by running MNIST with the Quickstart, then read Node and NodeGroup. The following NeuralNet and Tensor pages cover graph execution and data representation.
| What to change | Documentation and code locations |
|---|---|
| Model composition and connections | MNIST example, supported models, Writing custom models |
| Blocks shared by multiple models | Common groups, NodeGroup, Writing custom groups |
| Shape handling and GPU computation for an operation | Nodes and shaders, Writing custom nodes and kernels |
| Graph execution and buffer management | neural-net.cpp, Execution pipeline |
3. Common code conventions¶
3.1 CamelCase: Types start with an uppercase letter, while functions and variables start with a lowercase letter.
The rules for each construct are as follows.
| Construct | Rule | Example |
|---|---|---|
Classes, structs, enum, type aliases, concept |
PascalCase |
NeuralNet, PipeParams |
| Functions, member functions | camelCase |
loadWeights(), setProfileTag() |
| Variables, parameters | camelCase |
outShape, targetDtype |
public member variables |
camelCase |
acceptResidual, iDtype |
private, protected member variables |
camelCase followed by _ |
device_, cmdBuffer_ |
enum values |
PascalCase |
Fp16, Bilinear |
| Constants | camelCase, without a k prefix |
tokensPerSide |
| Namespaces | Lowercase | vai, node, detail |
| Macros | UPPER_CASE |
VAI_ASSERT |
The following names are exceptions.
- Tensor dimensions use the letters from their formulas. (Examples:
N,K,Co,Dh,Tmax.) - Configuration functions for model sizes start with
_because names cannot start with a digit. (Examples:_124M(),_3B_224().) - The
publicmemberconfig_ofNodehas a trailing_to distinguish it from the constructor parameterconfig.
3.2 Opening braces: The following rules apply to each construct.
| Construct | Opening brace |
|---|---|
| Regular functions, member functions | Next line (Allman) |
if, for, while, switch blocks |
Next line (Allman) |
| Struct and class declarations | Next line (Allman) |
Lambda functions, namespace, enum |
Same line |
namespace vai::node {
enum class Mode { Fast, Exact };
class Linear : public vai::Node
{
public:
void prepare()
{
if (config_.acceptResidual)
addResidual();
auto scaled = [&](uint32_t x) { return x * factor_; };
}
};
} // namespace vai::node
3.3 namespace: The engine uses the vai namespace, and common nodes and node groups use vai::node and vai::group.
Models and their own nodes and node groups go under model::<impl>::<model-name>. <impl> is production or experimental.
namespace vai
namespace vai::node
namespace vai::group
// For YOLOX
namespace model::experimental::yolox
namespace model::experimental::yolox::node
namespace model::experimental::yolox::group
3.4 Automated checks: Rules 3.1 and 3.2 are configured in .clang-format and .clang-tidy at the repository root and checked automatically on commits and PRs. Contribution workflow and automated checks covers the remaining formatting rules, such as line length and indentation, and how to pass the checks.