How do you detect drift when you have no labelled data?
Monitor proxies you can compute without labels. Embedding distributions of incoming requests, refusal and escalation rates, tool-call mix, retrieval score distributions, and answer length all shift before user-visible quality does. None of them tells you the system is wrong, but each tells you something changed and where to look.
Set the reference window during a period you are confident was healthy, then alert on distance from that window rather than on absolute thresholds. Absolute thresholds need tuning per feature and go stale; distance-from-reference generalises.
Pair the label-free signals with a small labelled canary set. The proxies tell you when to look; the canary tells you whether it actually got worse.
Written by Binary AI Labs · Reviewed