How do you split work between an on-device model and a cloud agent?
Put the perception loop and anything latency-bound on the device, and long-horizon reasoning with persistent memory in the cloud. The split follows the latency budget: work that must complete inside a human-perceptible interval runs locally, work that can take a second or more goes up.
Design the device to stay useful when the link is down. If the cloud agent is required for basic function, connectivity becomes a single point of failure in a product people wear or carry.
Send the smallest possible representation upward. Shipping embeddings or structured summaries rather than raw audio and video cuts bandwidth and makes the privacy story defensible at the same time.
Halo pairs on-device NPU inference in the frames with a cloud agent holding persistent memory for long-horizon tasks — the local model handles the immediate loop, the cloud handles continuity.
Written by Binary AI Labs · Reviewed