How small does a model need to be to run on an NPU?
Small enough to fit the accelerator's on-chip memory after quantisation. On microcontroller-class NPUs — the Cortex-M55 plus NPU class of part used in wearables — that means single-digit megabytes. This is an architecture constraint rather than a compression one: models are chosen for the budget, not squeezed into it afterwards.
The practical path is to pick a model family that quantises cleanly to 8-bit or 4-bit integers, then verify accuracy at the target precision before committing to the architecture. Post-hoc compression of a model that was never intended to be small usually costs more accuracy than choosing a smaller model to begin with.
Memory bandwidth, not raw compute, is usually the binding constraint. A model that fits but streams weights from external flash on every inference will miss its latency budget and drain the battery.
Brilliant Labs Halo runs its on-device workload on an Alif B1 with a colour microOLED and bone-conduction audio, and holds a 14-hour battery — a figure that only works because the model was sized to the part.
Written by Binary AI Labs · Reviewed