BigMoeOnEdge

The dense weights can now live in memory Android is not allowed to reclaim — worth +17.9% on a long generation.

Every token touches the dense (non-expert) weights, so they have to stay resident. Until now nothing could make them: RLIMIT_MEMLOCK is capped at 64 KiB by the vendor, the cgroup protections are v2-only, MGLRU is disabled at runtime, and MADV_COLD only redirects reclaim rather than preventing it.

There is one exception, and this release uses it. A dma-buf — allocated by an ordinary app through AHardwareBuffer — has pages that stay pinned for the buffer's lifetime, because a device may DMA from them at any moment. The kernel cannot take them.

--dense-weights ahwb

ahwb is the existing anon policy with a single substitution: the buffer comes from a locked AHardwareBuffer BLOB instead of the heap. The O_DIRECT read, the tensor rebind and the mmap handback are the same code, so an A/B against anon moves exactly one variable.

In the Android app it appears as Dense weights → Pinned (dma-buf).

Measured

In-app, Qwen3.6-35B-A3B-Q4_K_M, k=8, cache 3000 MiB, 1354-token generation, same session and same binary as its control:

| | anon | ahwb |
|---|---|---|
| decode | 2.588 tok/s | 3.053 tok/s (+17.9%) |
| bootstrap 95% CI | [2.497–2.687] | [2.928–3.176] — disjoint |
| dense_resident_frac | 0.848 | 1.000 |
| majflt / token | 265 | 257 — equal |
| compute_ms | 298 | 241 |
| io_ms, stall_ms, cache hit % | — | all unchanged |
| swap | 562 MiB | 294 MiB |

Reclaim-exemption holds: dense_resident_frac reads exactly 1.000, minimum included, in every pinned run.

The mechanism is not the one that was predicted

Major faults are equal between the two modes. anon already keeps the dense weights off the flash — that was never the gap. What it does not prevent is the kernel taking ~15% of them into zram, where a later touch costs a minor fault plus a decompression. That cost appears in no I/O counter and no fault counter, so it lands in compute_ms, which is a residual rather than a measurement of arithmetic.

Hence the entire delta showing up there while everything else stays flat.

anon protects the dense weights from flash. ahwb also protects them from zram, and the premium anon was quietly paying is worth about 18%.

The trade this was expected to lose on did not occur: the expert cache is untouched, hit rate identical to the decimal, because the dense set (~1.6 GiB here) is small next to a 3000 MiB cache budget.

The transferable part outlives the flag: compute_ms has been absorbing zram decompression all along, so earlier conclusions of the form "this regime is compute-bound" deserve re-examination.

Why it is off by default

• In the decisive pair ahwb ran first, and an order effect cannot be excluded. The reversed pair is owed.
• One device, one model, one configuration.
• Short turns cannot see it at all: three 67–74 token pairs were all inconclusive (per-token CV 33–71%, every interval overlapping). Reclaim is a standing condition that accumulates, so the gain needs a conversation-length generation to appear.

Turn it on in Settings if you run long turns on a model well past RAM; measure before trusting it elsewhere.

Also in this release

• bmoe-membench — a standalone probe for the read bandwidth of pinned memory, which is what gated the whole idea: gralloc decides per allocation whether a buffer is CPU-cacheable, and an uncached mapping would have read slower than the flash it replaces. It does not: a locked BLOB matches anonymous memory within 0.5% on both CPU clusters and at 4 threads.
• A hard limit worth knowing: AHardwareBuffer_lock fails with EINVAL at exactly 2^31 bytes — a signed 32-bit boundary — even though allocation reaches the 4 GiB format cap. The usable unit is 2047 MiB per buffer.
• Two negative results committed alongside the positive one: the inconclusive short-turn pairs, and a cross-day comparison reading +63.6% that is not usable, since anon alone moved +38.8% between the two days.

Full data: docs/bench-data/2026-07-21-pinned-dense-ab/ · docs/bench-data/2026-07-21-pinned-memory/ · CHANGELOG

Install

app-dev-release.apk below — sideload build, signed with the project's stable key, so it updates over previous sideloaded versions. Engine and app are unchanged for every mode other than the new one.