When should you run inference on-device instead of in the cloud?
Run on-device when latency has to be predictable, when connectivity is unreliable, or when the data should not leave the device at all. Outside those three cases the cloud is cheaper to run and far easier to update. On-device is a constraint you accept for a reason, not a default.
Predictable latency is the most common driver. A cloud round trip has a fat tail — the median is fine and the 99th percentile is not — and interaction loops that must feel immediate cannot absorb that tail.
Privacy is the second. If audio or video never leaves the device, the privacy story is structural rather than contractual, which is a materially different claim to make to users and regulators.
The cost is update velocity. Shipping a new model to devices in the field is a release, not a deploy, so the model you ship has to be right for longer.
Written by Binary AI Labs · Reviewed