Stanford and UC Berkeley researchers have introduced TokenGate, a new module that makes long-context LLM prefill up to 11.3x faster without accuracy loss. By using learned token pruning, TokenGate allows pretrained language models to skim inputs so that only essential tokens receive full processing.
The module adds a small multilayer perceptron gate and Low-Rank Adaptation to existing models like Qwen3 and Qwen3.5, enabling peak prefill memory reductions between 1.5x and 3.4x. Tested across tasks up to 128,000 tokens, the method outperforms multiple pruning baselines by utilizing a Mixture-of-Experts routing trick to make top-K token selection differentiable.
TokenGate provides an efficient, drop-in solution for lowering latency and memory barriers in long-context transformer applications. The source code and model checkpoints are publicly available on GitHub under an Apache-2.0 license.