Managing AI over time
Managing AI over time is the stage that never ends. It keeps every live assistant, brain and agent accurate, used and worth its cost, through regular checks, honest feedback, quarterly reviews and careful swaps to newer models.
Manage live AI systems with a rhythm: check exceptions daily, sample outputs weekly, audit quality monthly and review value quarterly. Watch usage, because low usage is an adoption problem. Collect feedback from the people using each system, test newer models against your own examples before switching and revisit the whole set every six to twelve months.
How do we manage AI systems once they are live?
Manage live AI systems with a fixed rhythm owned by the people who run each workflow: daily and weekly checks, usage monitoring, feedback, monthly quality audits and a quarterly review of value. Then revisit models and suppliers every six to twelve months.
Managing AI over time is the ongoing stage of the AI-Native Roadmap. It starts the day the first Phase 1 assistant goes live and runs alongside every later phase. Without it, accuracy drifts, usage fades and costs creep, usually without anyone noticing.
It is also how an organisation stays AI-native once it gets there. Hartz AI's definition, used across the roadmap: An AI-native business is one where AI assistants, department brains and agents do the repetitive work in every team, with people keeping the judgement.
What should we check every day and every week?
Check exceptions every day and sample outputs every week. Daily checks catch what has broken; weekly samples catch what has quietly got worse.
- Daily: failed automations, agent tasks passed to a person and anything a customer or member has queried.
- Weekly: a small sample of outputs from each system, read against the source, with errors logged by type.
At a primary school this might take ten minutes: the office manager glances at the overnight log each morning, and on Fridays reads five AI-drafted letters before the weekend. The point is that someone looks, on a schedule, every time.
How do we monitor usage?
Track how many people use each system, how often and for what. Low usage is the earliest warning that a system is failing, and it is an adoption problem rather than a technology one.
Confidence is often the reason. Government research by DSIT (opens in a new tab) found that only 21% of UK workers feel confident using AI tools at work. When usage drops, talk to the team before changing the tool: retraining, a clearer workflow or an AI champion in the team usually does more than a new product.
Watch for the opposite too. A sudden spike can mean a system is being used for something it was never approved for, such as personal data it should not see.
How do we collect feedback and audit quality?
Ask the people who use each system to rate it, and audit its outputs against the source every month. The two together tell you whether a system is trusted and whether it deserves to be.
Feedback loops
Give every system a simple way to flag a bad output, and run a short internal satisfaction score each quarter: how useful is it, how often is it wrong, what would make it better? A charity's volunteers are often the first to spot that a template has gone stale.
Quality audits
Each month, take a fixed set of real examples and check the system's answers against the source. The reason is in the research: in a Harvard Business School field experiment (opens in a new tab), knowledge workers using AI on a task chosen to sit outside the AI's capabilities were 19% less likely to produce correct solutions. Fluent output hides errors; audits find them.
What happens in a quarterly review?
A quarterly review asks one question of every live system: is it still worth what it costs? It looks at usage, quality, feedback and the business measure the system was built to move, then decides to keep, expand, fix or retire it.
Use is not value. In McKinsey's 2026 global survey (opens in a new tab), nearly nine in ten respondents report regular use of AI in at least one business function, yet only 37% attribute at least some impact on earnings before interest and tax to AI use.
The review is also where the next stage is chosen. A team whose assistant is well used and whose questions have outgrown it is ready for a department brain. A fractional Chief AI Officer can chair these reviews for organisations without a senior AI lead.
When should we pilot or swap to a newer AI model?
Swap to a newer model only when it beats the current one on your own tasks, measured against a fixed set of real examples. Pilot it alongside the old one first, and never change a model in a live workflow untested.
Prices fall fast, so it pays to look. The Stanford AI Index 2025 (opens in a new tab) reports that the inference cost for a system performing at the level of GPT-3.5 dropped over 280-fold between November 2022 and October 2024. A system built two years ago may now cost far more than it needs to.
A controlled swap keeps the risk low: run both models on the same test set, compare accuracy, cost and speed, switch one workflow, watch it for a fortnight, then switch the rest. Revisit the whole set of models and suppliers every six to twelve months.
What are the limits of AI today?
Today's AI systems have four limits worth managing around: how much they can read at once, how reliably they reason over many steps, their habit of stating wrong things fluently and the systems they cannot yet connect to.
- Context: a model can read only so much at once, so very long documents or histories need to be split or summarised first.
- Reasoning: long chains of steps go wrong more often than short ones, which is why Phase 3 keeps each agent's job narrow.
- Hallucination: models can invent facts, names and sources, so answers that matter are checked against the source.
- Missing connectors: some business systems have no way for AI to read from them yet, which limits what a brain or agent can see.
These limits shift with every model release, which is another reason for the six to twelve month review. The AI governance work records which limits matter for each system, and ongoing advisory keeps that record current.
Managing AI over time is the only stage of the AI-Native Roadmap that never finishes.
Common questions
Frequently Asked Questions
Keep your AI systems worth what they cost
Ongoing advisory runs the quarterly reviews, model checks and governance updates, so live systems stay accurate and used.