Measuring AI impact: knowing whether the step is worth keeping
In the previous lesson you turned AI use from a personal experiment into a written step inside a procedure, and it ended with one instruction: measure, then expand. This lesson answers the practical question behind it. What exactly do you measure, where do the numbers come from, and how do you decide whether the step deserves to continue, expand or stop? Without measurement the decision stays an opinion: one enthusiastic person says it saved a lot of time, another abandoned it after a single bad output in the first week, and nobody has evidence.
Record a baseline before you start
The biggest mistake in measuring any new tool is trying to measure after it is already running. At that point you have nothing to compare against, so you fall back on memory, and memory always exaggerates improvement. A week or two before you switch the step on, record the task as it is today: how many minutes it takes on average, how often it is redone or corrected, how many cases are completed per day. You do not need a measurement system; a simple sheet covering ten or twenty real cases gives you an honest starting point.
Four measures are enough at the start
Do not begin with a large dashboard. Four numbers cover most of what matters:
| Measure | What it tells you | How to collect it |
|---|---|---|
| Time per task | Whether you actually saved time | Start and end time for a sample of cases |
| First-pass acceptance rate | Output quality before human editing | Outputs accepted as-is divided by the total |
| Errors that reached the customer | The real risk to your reputation | Complaint log and later corrections |
| Cases completed | Whether capacity increased | A simple daily or weekly counter |
The first two measure efficiency, the third measures risk, the fourth measures capacity. If you look at time alone you may celebrate an apparent saving while errors quietly rise in the background.
A worked example
The numbers here are hypothetical and used only to illustrate the method. A services office receives written customer enquiries and has fixed a step that drafts the first reply, which an employee reviews before sending. Before switching it on they recorded twenty enquiries: the average time per reply and how many replies had to be rewritten. A month later they measured twenty more enquiries the same way.
What they are looking for is not the time alone. If time went down and the first-pass acceptance rate went down with it, the employee is now editing more, and the saved writing time simply moved into correction. If time went down while acceptance stayed steady and complaints did not rise, the gain is real. You always read two measures together, never one.
Calculate the net, not the gross
The visible saving is not the gain. Before you judge, subtract the real running costs: the tool subscription, the human review time you added, and the time spent training the team and updating the instruction whenever something changes. A step that saves an employee fifteen minutes but adds ten minutes of manager review is not necessarily a win, because the manager's time may be scarcer and more expensive. And an honest calculation keeps human review in place; removing it is not a saving, it is a transfer of risk to the customer.
A monthly checklist
- Do I have a baseline recorded before launch to compare against?
- Did I measure a sample rather than one memorable case?
- Did I read time together with quality, not time alone?
- Did I count review and training time as part of the cost?
- Did I check that errors did not simply move downstream instead of disappearing?
- Did I ask the person who runs the step daily, not only the manager?
Three decisions after measuring
Measurement should end in a clear decision. If efficiency improved and risk did not rise, expand: move the same cycle to one next task. If nothing improved, adjust before you blame the tool, because the cause is usually the written instruction or a poorly chosen task. If errors reaching customers went up, or review takes longer than doing the work manually, stop the step and record why. A decision to stop that rests on a number is a respectable one, and documenting it prevents the same experiment being repeated later.
Do not compare the tool against perfection. Compare it against what you were actually doing before. Your previous state is the benchmark, not an ideal that never existed.