What causes model drift in LLM applications?
Three things, and they look identical from the outside. The provider changes the model behind an unpinned endpoint. Your input distribution moves as real users find the product. Or a long series of small prompt edits accumulates into a behaviour nobody chose. Each produces the same symptom — quality falling without a deploy that explains it.
Telling them apart requires separate instrumentation. Version-pinned canary runs isolate provider drift. Embedding-distribution monitoring on inbound requests isolates data drift. Git history over prompt files plus per-commit eval scores isolates prompt drift.
In practice, data drift is the most common and the least monitored. A retrieval corpus that was representative at launch stops being representative once the product reaches an audience the launch cohort did not include.
Written by Binary AI Labs · Reviewed