All Posts
Blog
7
 min read

LLM Inference Optimization on AMD Strix Halo: 6.9× Faster Token Generation with yasp

Summary

Gaia, yasp’s model optimization agent, gets the most out of the hardware you already have by automatically tuning LLM inference for your model and machine. On AMD Strix Halo, compared with stock vLLM:

  • Up to 6.9× faster token generation: quantized Qwen3-14B (AWQ) went from 3.3 to 22.7 tokens per second.
  • Quantized LLMs finally pay off: with Gaia, 4-bit Qwen3-14B generates tokens up to 2.8× faster than the stock unquantized (BF16) model, where out of the box it was slower.
  • Long-context inference in a quarter of the time: with a 64k-token prompt on Qwen3.8-27B, time to first token (TTFT) falls from 7.8 to 2.2 minutes and end-to-end latency drops by 74%.
  • Better economics from existing hardware: Gaia’s generated kernels push the card to its memory-bandwidth limit, so the same machine does more work in less time, with no hardware upgrade.
  • No GPU performance engineers needed: Gaia’s agentic workflow profiles each model, finds its bottleneck and fixes it automatically, from serving configuration to custom GPU kernels.

‍

Three Qwen Models, One AMD Strix Halo, Three Different Bottlenecks

We took an AMD Strix Halo developer box and pointed Gaia at it. Gaia is yasp's model optimizer: it profiles how a model runs on your hardware, tunes the serving configuration, fixes execution paths inside the engine, and generates GPU kernels. Throughout this post, Gaia names that whole workflow: profile, serving configuration, engine execution-path fixes, kernel generation and the correctness gate, with generated kernels as the core of its output. The serving-configuration levers and the AWQ routing fix are not products of Gaia's kernel search; they are engine work, the same workflow drives and the reports label separately. This post shows what that means in practice across three Qwen3 models in a single vLLM stack on the Radeon 8060S GPU. Each model turned out to be held back by something different, and Gaia found and fixed a different bottleneck for each.

The headline results. Every row is stock vLLM as shipped against the best configuration of config plus kernels:

model format TTFT (s) decode tok/s
Qwen3-14B AWQ, 1,024-token promptAWQ0.99 → 1.003.3 → 22.7 (6.9×)
Qwen3-8B, 64,768-token promptBF16127.5 → 57.98.6 → 9.6 (1.1×)
Qwen3.8-27B, 65,536-token promptBF16468.9 → 129.91.03 → 4.15 (4.0×)

Every generated-kernel number below is measured against the same engine without the kernel, in the same serving session.

Decode speed: stock vs. Gaia-optimized (higher is better)

‍

The rig

Strix Halo with the Radeon 8060S and its unified-memory pool, vLLM 0.29.0 in a ROCm build, one GPU, one sequence, one request at a time. Every quoted claim runs 30 requests with 60 s warmup, GPU graph replay, prefix caching off (three 8B arms completed 29). The two long-context demonstrations referenced below use 3 requests each. Inter-token latency is measured from streamed tokens; request time is TTFT + 512 × inter-token latency. The workload is local, single-user inference, which is where Strix Halo and its unified memory land.

Example one: Qwen3-14B, choosing the format and building the kernels

Profiling the BF16 decode step showed 94–99% of the time streaming weights, at about 233 GB/s against 244 GB/s best observed on this card. Decode was already running at about 95% of the card's best bandwidth, so reducing bytes was the only large lever left. Out of the box, stock BF16 generated 8.1 tokens per second, while the quantized formats were slower: 3.3 for AWQ and 4.6 for FP8.

AWQ, 4-bit weights with fp16 activations, had the right answer but a wrong execution path. A Python branch picked dequantize-and-matmul over quantized compute during compiled decode, repeatedly expanding weights into memory traffic. Moving that decision into a custom operation took inter-token latency from 299.5 to 69.9 ms. A generated decoder-layer implementation then took it to 44.0 ms, another 37.1% off.

The quality price is real and should be on the table: AWQ's WikiText-2 perplexity is 9.128 against BF16's 8.742 (+4.41%), with FP8 at 8.773 (+0.35%); the correctness gate certifies each generated kernel against its own format's engine, not against BF16. FP8 rewarded hardware-specific tuning instead: the tested vLLM build shipped no tuned W8A8 tile config for this GPU, and fitting the launch configuration took FP8 from 216.1 to 82.0 ms (a cross-session comparison, like the AWQ headline; the kernel step that follows is same-session), with the generated implementation taking it to 65.4 ms.

Weight and KV-cache streaming rate, Qwen3-8B decode

‍

Example two: Qwen3-8B, long context

Native context is 32,768 positions and YaRN rope scaling extends the window to 131,072. On 32,768-, 64,768- and 130,304-token prompts, the serving configuration alone, AITER unified attention (AMD's ROCm AITER kernel library, github.com/ROCm/aiter) plus a 4,096-token prefill chunk, cut whole-request time 17–25% against stock BF16.

Two generated kernels then cut a further 11.4% (32k), 20.9% (64k) and 37.9% per request on top: a long-KV decode layer that streams weights and KV cache at 238–240 GB/s, the card's read ceiling, and a chunked-prefill attention kernel running at 41–44 TFLOPS where AITER runs 21–25. Against stock BF16 as shipped, the best configuration cut whole-request time 26.7% at 32k, 40.5% at 64k and 52.8% at 128k.

Qwen3-8B: decode speed by optimization stage

The kernel I would point at first is the long-KV decode layer. Decode at long context is a memory-streaming problem, and the generated layer treats the whole decoder layer as a single stream, reading model weights and the KV cache back to back. It holds 238–240 GB/s, the card's read ceiling, from 32k to 128k prompts, and it does it without re-reading or expanding anything, so the stream rate stays flat as the context grows.

Example three: Qwen3.8-27B, native rope, no YaRN

Configuration first: AITER unified attention plus a 1,024-token prefill chunk with skinny GEMM off cut whole-request time 52.2% at 32k and 60.5% at 64k against stock BF16. That is two engine arguments plus one environment variable, and it delivered more than half the total gain. Two generated attention kernels, certified at 32,768 and 65,536 tokens, cut a further 18.1% and 33.7% against that configuration. Whole-request time against stock fell 60.9% at 32k and 73.8% at 64k, and time to first token fell from 468.9 to 129.9 s at 65,536 tokens..

Qwen3.8-27B BF16: decode speed across context lengths

The config lever mattered here: the stock attention backend read the KV cache at 2.4% of the card's best rate (profiled at 8,192 tokens). That configuration work is part of the same optimization workflow the kernels come from; a kernel generator that ignores the serving stack leaves most of the win on the table.

The correctness gate

Generated kernels are certified by an in-engine gate: full-vocabulary distribution comparison against a clean reference, the same engine with no kernel loaded under the served configuration, over dependent decoding steps, with deliberately incorrect negative controls that must fail.

Generated kernels are checked against the corresponding unmodified engine; certification applies only to the kernel and context lengths listed below:

  • Qwen3-14B: the AWQ decoder-layer kernel and the FP8 decoder-layer kernel are certified at 1,024, 4,096 and 8,192 tokens against their own format’s unmodified engine on vLLM 0.29.0+strix.
  • Qwen3-8B: the long-context decode kernel and the decode+prefill pair are certified at 32k and 64k, not at the 130,304-token (128k) measurement.
  • Qwen3.8-27B: the two attention kernels are certified at 32,768 and 65,536 tokens.

‍

Key Takeaways: The Profile Decides

Three models on one hardware platform needed three different fixes:

  • Qwen3-14B: the key was choosing a smaller number format and fixing how the engine executed it, followed by generated kernels. Token generation in AWQ became 6.9× faster.
  • Qwen3-8B: a tuned serving configuration plus two generated kernels cut end-to-end latency by up to 53% at 128k tokens. The decode kernel runs at the card’s memory-bandwidth ceiling, and the prefill attention kernel reaches 41–44 TFLOPS, compared with 21–25 for the AITER baseline.
  • Qwen3.8-27B: The configuration delivered most of the time saved (86% at 32k, 82% at 64k), and generated kernels added the rest, for 74% lower latency at 64k tokens.

Four lessons stand out:

  • Stock is not the ceiling. Out of the box, quantized and long-context workloads on AMD Strix Halo leave much of the hardware’s performance unused.
  • There is no single fix. The right lever was different for every model, which is why Gaia profiles first and optimizes second.
  • The whole stack matters. Serving configuration, engine execution paths and kernels all contributed. A kernel generator that ignores the serving stack leaves most of the win on the table.
  • Speed without guesswork. Generated kernels are checked against the corresponding unmodified engine; certification applies only to the kernel and context lengths listed in the correctness section. Quantization tradeoffs are measured and reported, not hidden.
    ‍

What this means for teams running AI on AMD

The same machine does more work in less time, without a hardware upgrade. In our tests, the quantized model went from slower than full precision to the fastest option. Long-context tasks, such as analyzing long reports or large codebases, become practical on local hardware.

Getting there normally requires scarce GPU performance engineers tuning each model for each chip. Gaia automates that work. It optimizes your model on your hardware for your workload, from the serving configuration down to the code running on the GPU.

What’s next

With BF16 models, token generation on Strix Halo is now limited by memory bandwidth, so the next gains come from moving fewer bytes: lower-precision formats and speculative decoding.

Optimize your model on your hardware with yasp

Gaia brings the same workflow to your model and your AMD machine. Want to run it yourself? Request the optimized model, with the generated kernels for AMD Strix Halo, ready to use with vLLM.

‍

Figure 1: Pipeline utilization of the PyTorch Inductor solution

See gaia & kronos in action

Bring your model. 30 minutes. No slides.

Get Started
Get Started
Share