Build ik_llama.cpp with AVX512 on Windows

For this blog is about an experimenting with building ik_llama.cpp, I was wondering how fast AVX512 on a Ryzen 9 9700X + 5070 Ti would be if enabled. I've also documented the setup and a how-to guide on my blog.

Install Tools For Build

📌C++ Build Tools

Since I already use Visual Studio for C# ASP.NET development, I chose to install C++ directly from the Visual Studio Installer. The additional components I installed include:

Actually, you can just install the standalone Build Tools Build Tools for Visual Studio 2026 - the individual components same as vs2026 or WinLibs-build of GCC MinGW-w64

📌 NVIDIA CUDA Toolkit - a development environment for creating high-performance, GPU-accelerated applications Now Shift with CUDA13.xx

Build

From Doc https://github.com/ikawrakow/ik_llama.cpp/blob/main/docs/build.md I recap step here

cd C:\llamaCPP\ik_llama.cpp

mkdir build

cd build

📌 Build Step - Open x64 Native Tools Command Prompt for VS is recommended because it injects the relevant environment variables for C++ Build

cmake .. -DCMAKE_BUILD_TYPE=Release -DGGML_AVX512=ON -DGGML_AVX512_VBMI=ON -DGGML_AVX512_VNNI=ON -DGGML_NATIVE=OFF -DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES="80;86;89;90;120"

From cmake command, this command configures a CMake build for ik_llama.cpp (a llama.cpp fork), targeting AVX-512 CPU optimization and NVIDIA CUDA GPU support.

📌 Build type

  • -DCMAKE_BUILD_TYPE=Release — compiles with optimizations enabled (vs. a debug build), which is what you want for actual inference performance.

📌 CPU: AVX-512 flags

  • -DGGML_AVX512=ON — enables AVX-512 codegen. On MSVC this maps to /arch:AVX512; on GCC/Clang it adds the equivalent -mavx512f style flags. This defines the __AVX512F__, __AVX512VL__, __AVX512BW__, __AVX512DQ__, __AVX512CD__ macros the codebase checks
  • -DGGML_AVX512_VBMI=ON and -DGGML_AVX512_VNNI=ON — enable two specific AVX-512 sub-extensions. VNNI in particular is important: the IQK quantized GEMM kernels (the hot path for quantized prompt processing) are gated behind a HAVE_FANCY_SIMD macro that requires AVX512F + AVX512VNNI + AVX512VL + AVX512BW + AVX512DQ all being defined at compile time. Miss one, and the build silently falls back to slower AVX2 — no warning, no error, just lower throughput.
  • Notably, GGML_AVX512_BF16 is not set here (the docs' recommended snippet includes it) — so native bf16 GEMM support won't be compiled in. If your CPU doesn't support AVX512_BF16 anyway, that's a non-issue; otherwise it's a flag you might be leaving on the table.

📌For Portable Binary DGGML_NATIVE=OFF

  • -DGGML_NATIVE=OFF - a more portable binary that only uses the instruction sets you explicitly enabled not fixed with HW (CPU/ GPU)
  • if you want a best performance -DGGML_NATIVE=ON / GGML_NATIVE=ON , but build machine HW must same set as inference HW

📌GPU: CUDA

  • -DGGML_CUDA=ON — enables GPU acceleration using the CUDA cores of your Nvidia GPU, requires the CUDA toolkit installed.
  • -DCMAKE_CUDA_ARCHITECTURES="80;86;89;90;120" — tells nvcc which GPU compute capabilities to generate PTX/SASS for, so the binary works across a range of NVIDIA generations without needing to rebuild per-GPU:
    • 80 — A100 (Ampere)
    • 86 — RTX 30-series, A40 (Ampere)
    • 89 — RTX 40-series, L40 (Ada Lovelace)
    • 90 — H100 (Hopper)
    • 120 — RTX 50-series / Blackwell consumer

Next - Build from configures a CMake, By using command 

cmake --build . --config Release -j 16
High CPU Usage. - When cmake --build . --config Release -j 16

Package CUDA

  • copy cuda dll C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v13.3\bin\x64 to C:\llamaCPP\ik_llama.cpp\build folder

Run on Target Machine

  • zip folder C:\llamaCPP\ik_llama.cpp\build
  • run on target machine 
.\llama-cli.exe -m "C:\Users\User.lmstudio\models\unsloth\Qwen3.8-27B-GGUF\Qwen3.8-27B-UD-Q3_K_XL.gguf" -ngl 99 -t 16 -p "Write a Code FB Login page" -c 40000

Here is a preliminary test. From the image above, you can see that the prefill speed increased significantly from 18.39 to 56.43 Tok/sec. Meanwhile, the output generation phase only increased slightly to about 45 Tok/sec. However, compared to running the same model using LM Studio (15-20 Tok/sec), the speed is significantly better.

If I do any further testing, I'll write another blog post about it.

Reference

Tags

Related Posts