What NIST's guidance on monitoring deployed AI means for M&E teams
Oct 27 2026 · Governance
NIST treats post-deployment AI oversight as continuous evaluation. Here is how M&E teams can turn that into indicators, cadence and named responsibility.
NIST's guidance on monitoring deployed AI systems frames post-deployment oversight as a continuous evaluation task: detect drift, keep a human in the loop, and document what you watch and why. For M&E teams, the practical takeaway is that monitoring AI output is not a new discipline. It is an evaluation problem. Set a baseline, choose indicators, decide a cadence, and name the person who acts on a signal. The open questions in the guidance, namely what to monitor, how often, and who is responsible, are the same questions a logframe or evaluation plan already forces you to answer.
What does NIST's guidance on monitoring deployed AI actually say?
The guidance treats monitoring as a continuous activity that starts after deployment, not a one-off test before launch. Its central concerns are drift, human oversight, and the gap between automated checks and human judgement.
It also leaves questions open: what to monitor, how often, and who is accountable. Those are evaluation questions, and M&E teams already have the vocabulary for them.
Why does AI monitoring belong on an M&E desk?
Because the work is the same shape. You define what good looks like, choose indicators that can actually be measured, set a cadence, and name who acts when a number moves.
Most AI governance documents stop at principles. Monitoring is where principles become observable, and observation is what M&E does. If you already maintain a logframe (/blog/logframe-template-google-sheets), you are closer to an AI monitoring plan than most engineering teams.
What should you monitor in a deployed AI system?
Pick a small set of indicators you can compute repeatedly. Four cover most cases.
- Pass rate on a fixed golden set of inputs, scored the same way each time.
- Hallucination rate: outputs that assert something the source material does not support.
- Citation accuracy: whether a cited source exists and says what the output claims.
- False-confidence rate: how often a wrong answer is delivered in a confident tone.
Track these against a baseline recorded before launch. Without a baseline you cannot tell drift from noise. The indicator design guidance at /store/indicator walks through the same steps for programme indicators.
How often should you monitor, and who acts on the signal?
A monthly review is a workable default for most teams. Add an immediate check after any model update, prompt change, or data source change, because those are the moments when behaviour shifts fastest.
Cadence is only half the answer. Name one person who owns the signal and one action they are authorised to take. Monitoring without an owner produces dashboards nobody reads.
Monitoring is not the dashboard. It is the decision that follows the dashboard. If no one is authorised to change the prompt, the rules file, or the model, you have measurement without control.
How is AI monitoring different from traditional M&E?
| Dimension | Traditional programme M&E | Monitoring a deployed AI system |
|---|---|---|
| What you observe | People, households, sites | Model outputs on real inputs |
| Baseline | Pre-intervention measurement | Golden set scored before launch |
| Cadence | Quarterly or annual | Continuous, with scheduled reviews |
| Typical failure | Activity happened, outcome did not | Output reads well, is wrong |
| Who acts | Programme manager | Named owner of the AI system |
The differences are real but shallow. The underlying discipline of define, measure, review and act carries over intact.
How do you write an AI monitoring plan?
- Write down what the system is for and what a wrong output would cost.
- Record a baseline on a fixed golden set before launch.
- Choose three to five indicators and define how each is scored.
- Set a review cadence and a trigger for out-of-cycle checks.
- Name the owner of each indicator and the action they can take.
- Decide what evidence triggers a change to the prompt, rules file, or model.
- Review the plan itself once a quarter and delete indicators that never move.
A structured evaluation plan (/store/evalplan) gives you the scaffolding. The AI-specific parts are the golden set and the update triggers.
What warning signs mean your monitoring is not working?
- Nobody has looked at the indicators since launch.
- The golden set has not changed while the inputs have.
- Every indicator is green because they are all self-reported by the vendor.
- A wrong output reached a user and no one can say which check should have caught it.
- The review meeting has no decision recorded against it.
Any one of these means the plan exists on paper only. The fix is usually an owner, not more indicators.
Where do you start?
Start with the golden set. Twenty to fifty representative inputs, scored by hand, stored somewhere you can rerun them. That single artefact turns an abstract governance commitment into something you can measure next month.
If you are still choosing a model or tool, /which-ai compares options against evaluation needs. If you need the wider toolkit, /tools lists what is available.
Written for M&E professionals. Not a guarantee of accuracy — always verify every claim against its source, and let a named human sign off. Get the free handbook.