How do you stop an LLM agent from drifting in production?
You cannot prevent drift — you detect it. Pin model versions, score a fixed canary set on every deploy and daily, and alert on the delta rather than the absolute score.
Read the full answer →ANSWERS
5 ANSWERSModel drift
Why systems that passed review degrade in production, and what catches it.
What causes model drift in LLM applications?
Three sources: the provider changing the model, your input distribution moving, and accumulated prompt edits. They look identical in the metrics and need different fixes.
Read the full answer →How often should you re-evaluate a model in production?
A fast subset on every deploy, the full suite nightly, and a human-reviewed sample weekly. Anything slower and you learn about regressions from users.
Read the full answer →How do you detect drift when you have no labelled data?
Monitor the inputs and the system's own behaviour: embedding distributions, refusal and escalation rates, tool-call mix, and answer length. All are label-free and move before quality does.
Read the full answer →Should you pin model versions or take automatic upgrades?
Pin, then upgrade deliberately behind an eval run. Automatic upgrades trade a small maintenance saving for an unbounded, unobservable quality risk.
Read the full answer →