Zyphra has released Zyphra Hierarchical Attention (ZHA), a sparse attention method built around a tree-structured indexer designed to achieve true subquadratic attention. Traditional sparse attention methods often suffer from a significant bottleneck where the indexer that picks tokens still scans the entire context, consuming up to 90 percent of layer time near one million tokens. ZHA addresses this gap by making the token search itself sparse, keeping the full key-value cache while using a tree structure to decide which cached keys each query reads.
According to Zyphra's technical paper, ZHA can be efficiently distilled into frozen pretrained models using about four billion tokens. When tested on the Qwen3.6-35B-A3B model, ZHA stays close to dense attention performance across reasoning, agentic coding, and 256K retrieval benchmarks. The method utilizes a top-p routing mechanism to make the token budget adaptive, allowing queries to dynamically expand tree branches based on the information needed. On the LongBench v2 benchmark, the deep-tree configuration scored 63.4 percent compared to 58.0 percent for dense attention, while using only about 1.1 percent of keys.
While speedups at a 256K context are modest, modeled projections based on H100 BF16 rooflines suggest significant performance gains at much larger scales, such as millions of tokens. This development provides a scalable path forward for handling extremely long contexts in transformer models without requiring fully dense attention computations.