Why an agent harness matters in HIP kernel generation

Harness engineering is becoming a core part of agent design. Garry Tan, Y Combinator's president and CEO, said that 2027 would be the year of the "Harness War". At yasp, we have invested in harness design from the beginning. Gaia is our model-optimization and kernel-generation agent, built around one goal: any model, any hardware. In this article we focus on how Gaia generates optimized kernels for AMD hardware.

Let's give freedom back to builders. Builders provide a reference implementation and specify that hardware. Gaia generates, evaluates, benchmarks, profiles, and tunes kernels for the target hardware through a simple command-line workflow. Gaia is designed to complete the optimization loop autonomously and return an optimized kernel rather than function as an interactive coding assistant.

yasp-toolkit kernelgen compile \
  --reference /path/to/reference-kernel \
  --gpu-name "$GPU" \
  --output /path/to/optimized-kernel \
  --report /path/to/report.json

The PyTorch-to-HIP challenge

AMD GPUs, including both Radeon and Instinct GPUs, are important for AI workloads but remain underutilized. According to the AgentKernelArena paper, PyTorch-to-HIP is one of the most demanding kernel-generation workloads for agents, as they must translate PyTorch models into complete HIP implementations from scratch. This difficulty prevents some agents, such as GEAK, from competing in the PyTorch-to-HIP challenge. In the AgentKernelArena study, PyTorch-to-HIP challenge is a set of 24 problems written in PyTorch. The agents are evaluated on several factors such as compilation, correctness across multiple input shapes, and performance relative to eager PyTorch.

Results

In this study, the model for Gaia uses Opus 5 (high effort) as its foundation model. The baselines for the PyTorch-to-HIP challenge in the AgentKernelArena paper used the foundation models available at the time of publication. To keep the comparison fair, we re-ran Claude Code with the current model rather than only citing the published numbers. The following table compares Gaia against Claude Code running Opus 5 Max as well as the numbers reported by the AgentKernelArena paper.

Gaia and the Opus 5 Max re-run were executed on a DigitalOcean MI300X instance while all other configuration follows the paper. Remaining rows are quoted from the paper.

On the benchmark's own composite scale, introduced by AMD, 20 points for compilation, 100 points for correctness, plus 100 points × speedup, Gaia averages 1750.9 points against 1308.3 points for Claude Code, which given identical compilation and correctness is a restatement of the speedup gap.

The headline number for Gaia is a 16.31× average speedup compared with 11.88× for Claude Code using the paper's methodology. The difference is concentrated in the larger and more complex kernel-generation workloads such as attention kernels, where Gaia found a different algorithm. The agent moved beyond fusion and changed the algorithm itself. In Normal Attention case Gaia identified the low-rank structure of the attention scores and avoided constructing the full attention matrix. That reduced the intermediate memory traffic significantly which explains why these two workloads reached 2.52× and 3.02× the speedups achieved by Claude Code.

It is worth noting that throughout the experiment, Gaia used its default reasoning level (High) while the Claude Code configuration published in the Agent Kernel Arena GitHub repository sets the reasoning level to Max. Max is the highest reasoning level, the result: Claude Code generates kernels at a much higher cost than Gaia.

Why the harness makes the difference

Let's explore Gaia's results from a different angle, in this experiment compared with those of the other agents. Cursor Agent and Claude Code are both general-purpose coding agents, while Gaia is an expert in model optimization and kernel generation. The way Gaia works is that it takes the reference kernel, much like a compiler turns the input into an intermediate representation called yaspIR. It then routes that representation to a hardware-specific backend agent which has knowledge bases on optimization strategies and target architectures.

Evaluation is a required node in Gaia's execution graph rather than an optional tool call. Each candidate passes through compilation, correctness validation, reward-hacking checks, benchmarking, and profiling. The resulting errors and performance measurements are fed back into the generation loop, allowing Gaia to revise the implementation and generate a new candidate. In this run, that process produced kernels that compiled and passed correctness checks for all 24 tasks, with a 16.31× mean speedup.

It is worth examining why Gaia succeeded on the PyTorch-to-HIP task while other kernel generators might not. According to AgentKernelArena, generating HIP kernels often fails either on unseen shapes or generating kernels from scratch. These are Gaia's key differentiators. Gaia handles dynamic shapes like a compiler and ensures that we obtain the best candidate kernel for the range of data the user expects. Then, Gaia uses its knowledge base to generate a hardware specific kernel (HIP for AMD, NKI for Trainium, etc.) from scratch.

Try it on your own model

Feel free to reach out to me at reza@yasp.ai, if you have any questions or you want to evaluate Gaia on your own model or target hardware. You can also directly book a demo at book-a-demo.

‍