When should you run inference on-device instead of in the cloud?
When latency must be predictable, connectivity is unreliable, or the data should not leave the device. Otherwise the cloud is cheaper and easier to update.
Read the full answer →ANSWERS
4 ANSWERSEdge inference
Running models on-device — latency, power, and the trade-offs against cloud.
How small does a model need to be to run on an NPU?
Small enough to fit the NPU's on-chip memory after quantisation — on microcontroller-class parts that means single-digit megabytes, which is an architecture constraint, not a compression one.
Read the full answer →How do you split work between an on-device model and a cloud agent?
On-device handles the perception loop and anything latency-bound; the cloud handles long-horizon reasoning and persistent memory. The split follows the latency budget.
Read the full answer →What latency is achievable for on-device computer vision?
Sub-10 ms per frame for vision-transformer inference on current mobile and embedded accelerators, which is enough for a real-time interaction loop.
Read the full answer →