Developer Vixhal has released qwenGPU, an experimental model that compiles a SIMT GPU directly into the weights of Qwen3.5-2B under the MIT license. The model runs real CUDA, compiled to NVIDIA PTX, inside its own forward passes to test exact computation inside a pretrained large language model rather than beside it.
The architecture combines Qwen3.5's 24 layers with 45 compiled transformer layers, totaling 69 layers with a 262,144 token context. The integrated 4-lane SIMT GPU features 15 registers per lane, a 4-stage pipeline, a 32-block ROM, and 126 memory words, utilizing Gated DeltaNet linear attention and SwiGLU MLPs as memory cells. The host compiles CUDA to PTX with clang, feeding the program and input arrays to the model as tokens where execution outputs return via recurrent state records.
Running the model requires about 16 GB of free memory along with Python 3.11, uv, and just, while advanced mode also requires clang with the NVPTX backend. Benchmarks show the reference interpreter matches an NVIDIA Tesla T4 across 49 test kernels while maintaining a text distribution KL divergence of approximately 7e-4 from the original Qwen3.5-2B model.
This project matters because it demonstrates the feasibility of embedding precise, executable hardware simulation directly into the weights of standard pretrained language models.