How yasp found the optimization in IBM Granite 4.0 that no engineer had time to find

A single algebraic insight, found automatically, eliminated 48 GB of wasted GPU memory and made the Mamba layer 6.25x faster.

Published on

Jul 2026

Summary

IBM's Granite 4.0 uses a Mamba hybrid architecture that standard compilers can't optimize. yasp's agentic compiler identified that a memory-bound operation was algebraically equivalent to a matrix multiplication, eliminated a ~48 GB intermediate tensor, and achieved a 6.25x layer-level speedup on NVIDIA H200 — with up to 3x end-to-end model improvement.

Impact

Up to 3x

End-to-end model speedup

6.25x

Faster Mamba layer inference

"Granite 4.0's Mamba layer was creating a 48 GB temporary tensor every forward pass, only to immediately discard it. Our agents recognized the multiply-then-sum was algebraically equivalent to a batched matrix multiply. One rewrite, applied in four places, took the layer from 22.6ms to 0.65ms. Thread blocks went from 6.3 million to 6,144. Every output verified identical."

Christian Leibig

Lead Agentic AI

,

yasp

Challenge

Granite 4.0 uses a Mamba hybrid architecture — efficient in theory, but standard compilers weren't built for it. PyTorch's Inductor backend knows how to optimize transformer patterns. Mamba's computational patterns are different. The compiler doesn't recognize them and generates code that wastes orders of magnitude more memory than necessary.

Deep inside the Mamba layer, one operation was creating a ~48 GB temporary structure every forward pass — 12.9 billion numbers — only to immediately discard it. The operation took 22ms. The rest of the layer was faster than this single step. It wasn't a bug — it was a legitimate multiply-then-sum that standard compilers handle the naive way.

Solution

yasp's agentic compiler identified that the multiply-then-sum was mathematically equivalent to a standard matrix multiplication — an operation GPUs execute at peak efficiency. The compiler rewrote the operation, eliminating the 48 GB intermediate entirely. The same pattern appeared in four places within the layer. yasp found and rewrote all of them.

The layer-level improvement was 6.25x. Across the full model, end-to-end inference was up to 3x faster — all outputs verified identical.

Impact

Thread blocks dropped 1024x. The GPU shifted from address arithmetic to actual computation. No accuracy trade-off — verified via torch.allclose. The model designed to be efficient became actually efficient. And the same approach works on the next architecture that doesn't have a name yet.