Getting large language models to run on a phone is no longer a research experiment. It is a shipping reality. Yet the moment you move from a toy demo to an actual product, latency reasserts itself. You trimmed the model, quantized the weights, and still the prefill phase drags. The tokens stall. The UI freezes. The blame rarely lands where it should.
On Android devices, compute is almost never the bottleneck during LLM prefill. Memory bandwidth is. Modern flagship SoCs ship with powerful GPU and NPU cores that can burn through arithmetic far faster than the memory subsystem can feed them. When you profile a naive attention implementation, the execution units are not saturated. They are waiting. Waiting for DRAM.
Why Your Quantized Model Still Feels Slow
Quantization has become the default first step for on-device inference. Shrinking weights from FP16 to INT8 halves model size and cuts storage. It helps. But it does not fix attention layer latency. The reason is simple: quantization reduces the amount of data you store, but it does not reduce the number of memory transactions the attention mechanism performs.
A standard multi-head attention layer, implemented the textbook way, makes three complete round trips to DRAM for every layer. Query, Key, and Value matrices are read from main memory, scores are computed, and intermediate results are written back out. The arithmetic is trivial. The data movement is brutal. On Android, where power and thermal budgets are tight, this pattern thrashes the memory bus. The processor is effectively paying toll three times to cross the same bridge.
If you have been shipping INT8 models and wondering why the prefill step still scales quadratically with prompt length, this is the answer. The weights are smaller, but the activation traffic remains enormous.
The Bottleneck Is Memory, Not Math
To understand the fix, look at the roofline. Mobile GPUs and NPUs on chips like the Snapdragon 8 Gen 3 and Dimensity 9300 have theoretical compute throughput that far exceeds what their LPDDR5X interfaces can sustain. In a naive attention kernel, each head computes softmax over the product of Q and K, then multiplies by V. Every intermediate score matrix gets materialized in global memory. That means your peak DRAM reads scale with the square of sequence length, or O(n²). For a 1024-token prompt, the memory traffic is already large enough to dominate execution time.
The cores are underutilized because they cannot hide latency. Modern processors rely on caches to keep pipelines full. When an algorithm constantly misses in cache and fetches from DRAM, the execution units sit idle. No amount of quantization fixes this structural mismatch between compute capacity and memory supply.
How Tiling Reclaims Bandwidth
The solution is a tiling strategy that keeps intermediate scores in on-chip SRAM instead of shipping them to DRAM. This is the same insight that drives Flash Attention, adapted for Android's compute stack. Rather than materializing a full n × n score matrix in memory, you break the computation into small tiles that fit inside L1 cache. You compute local softmax statistics, accumulate running max values and normalization sums, and only write the final weighted outputs back to memory.
This changes the bandwidth complexity. Peak DRAM reads drop from O(n²) to O(n), because you no longer need to shuttle full score matrices through main memory. The heavy lifting happens inside SRAM, right next to the execution units.
For a concrete example, consider a tile size of 64 and a head dimension of 128. The score tile occupies 16 KB. That footprint sits comfortably inside the L1 cache of current flagship SoCs like the Snapdragon 8 Gen 3 and Dimensity 9300. The arithmetic stays local. The memory bus breathes.
Implementing This on Android
The algorithmic sketch is straightforward, though getting the details right matters.
Break your Query, Key