New AI model releases: what should M&E teams actually do?
Oct 13 2026 · Models
Model releases are a trigger to re-test, not to migrate. Here is the golden set, the model log and the routine that keep M&E outputs defensible.
Treat every new model release as a trigger to re-test, not a trigger to migrate. The model you rely on can change behaviour — tone, length, and how readily it fills gaps — without any change to your prompts. So the practical response is to re-run your golden set: a small batch of M&E tasks with known correct answers, such as summarising a donor report or extracting indicators from a logframe. Compare the scores to your last run, tighten the rules file where quality dropped, and log the model version, date and results. Only switch tools or models when a task's test scores justify it, and never mid-evaluation.
What should M&E teams do when a new AI model launches?
Treat it as a maintenance event, not a migration. The release itself is not the risk. The risk is that a model you already rely on has changed how it writes, how long it writes, and how willing it is to guess when a document is incomplete.
So the first action is a re-run, not a purchase. Take the small set of M&E tasks you have known-good answers for, run them again, and compare. Everything else follows from that comparison.
Why does a new model change results you already signed off?
Prompts are not contracts. They are inputs to a system that is updated on someone else's schedule. A prompt that produced a careful, sourced summary last quarter can produce a smoother, more confident one this quarter without a single character changing on your side.
For evaluation work the sensitive areas are predictable: invented baselines, dropped caveats, hedged language turned into flat assertion, and category labels that quietly shift between batches. None of these arrive as an error message. They show up later, in a report.
Which model should you use for which M&E task?
Most teams do not need one winner. They need a mapping from task to model, written down and reviewed when releases land.
| Task | What to look for | What goes wrong when it drifts |
|---|---|---|
| Summarising donor and programme reports | Keeps structure, points to sections | Caveats and limitations quietly dropped |
| Extracting indicators from a logframe | Returns exact wording, flags blanks | Invents a baseline or a missing year |
| Drafting survey and interview questions | Plain language, no leading phrasing | Subtly leading wording enters the tool |
| Translating tools and consent forms | Consistent terminology across rounds | Register and tone drift between rounds |
| Coding open-ended responses | Stable categories across batches | Category boundaries move mid-dataset |
If you are choosing between tools rather than models, the Which AI? guide at /which-ai works through the same trade-offs for M&E practice.
How do you re-test without losing a week?
- Pick ten to twenty tasks you already know the right answer for.
- Run them on the current model with your existing prompt and rules file.
- Score each one against your known answer: correct, partly correct, wrong, or invented.
- Compare to your previous run and note which category moved.
- Tighten the rules file only where the drop is real.
- Re-run the changed tasks to confirm the fix, not the whole set.
- Log the model version, date, prompt version and scores.
- Leave everything else alone until the next release.
What belongs in a golden set?
A golden set is small on purpose. It is the handful of tasks you can check by eye, quickly, because you already know what a good answer looks like.
- One summary of a real report where you know which limitations were stated.
- One indicator extraction from a logframe with a deliberately missing value.
- One translation of a consent form or data collection tool where terminology matters.
- One batch of open-ended responses you have already coded by hand.
- One task that should make the model refuse or ask a question rather than answer.
The last one matters most. Models that have become newly confident tend to fail there first. For the logframe task, the Google Sheets logframe template at /blog/logframe-template-google-sheets is a practical place to keep the known-good extraction next to the source.
What should you record so your results stay defensible?
An evaluation that used an AI tool needs to be explainable to a commissioner, an ethics committee or an auditor. That means knowing which model produced which output, and when.
Record the model name and version, the date of each run, the prompt version, and the task it was used for. If a tool does not expose its model version, say so in the limitations section. Three lines in a methods annex saves a long conversation later.
Should you switch tools when a new model arrives?
Usually not immediately. If your current tool still scores the same on your golden set, and the new option does not offer something specific you need — better handling of scanned documents, offline processing, data residency — switching costs you retraining, rewritten templates and re-verification of every prompt.
The exception is a task that is currently failing and a new option that fixes it. Test that one task before moving anything else. If you are building or rebuilding an evaluation plan, the templates at /store/evalplan and the indicator library at /store/indicator are a faster starting point than a new tool. Teams doing this often keep a shared library of prompts and rules rather than rebuilding each time, which is what /store/vault is for.
What does a monthly routine look like?
- Put a check in the diary once a month, or whenever a release reaches a model you use.
- Re-run the golden set. Fifteen minutes is usually enough.
- Update the model log with the date and scores.
- Fix only what regressed.
- Review your task-to-model mapping once a quarter, not on every release.
The teams that handle this well are not the ones who chase every launch. They are the ones with a short test they actually run. /tools lists what is in the library, /membership covers the prompt sets when they are updated, and /book is where to start if you would rather map your tasks with someone.
Written for M&E professionals. Not a guarantee of accuracy — always verify every claim against its source, and let a named human sign off. Get the free handbook.