Should you pin model versions or take automatic upgrades?
Pin, and treat every upgrade as a deploy with its own eval run. Automatic upgrades save a little maintenance and buy an unbounded quality risk you cannot observe. The saving is real but small; the risk lands on a day you did not choose.
A sane upgrade path: pin production, run the new version against the full eval suite in a staging environment, compare scores per scenario cluster, then ship. If any cluster regresses beyond the budget, the prompt needs work before the model moves.
Keep the old version reachable for one release cycle so a rollback is a config change rather than an incident.
The exception is genuine capability jumps, where the new model is better across every cluster. Even then, the eval run is what tells you it is — not the release notes.
Written by Binary AI Labs · Reviewed