Measuring the value of AI: indicators, starting point and pitfalls to avoid

In a finance department, a private equity fund, a family office or an accounting firm, artificial intelligence projects are multiplying. A tool that helps draft analysis notes, automatic sorting of invoices, an assistant that prepares answers to a due diligence questionnaire, that is, the in-depth review of a target before an investment is made, the extraction of clauses from contracts. Teams are motivated, the demonstrations are convincing, the early feedback is enthusiastic. Then comes the question every executive eventually hears in committee: what has this actually delivered? All too often, the answer remains vague, because nobody measured anything before the launch.

This difficulty is not unique to AI, but AI amplifies it. Management software replaces a specific step whose cost is known. A language model, that is, software capable of producing, summarising or classifying text from written instructions, tends instead to come into play across a multitude of scattered intellectual tasks, whose duration varies from one person to another and from one case to another. The gain, where it exists, is often diffuse, sometimes invisible in the accounts, and sometimes overestimated by those who want to believe in it. To find one's bearings, it is best to start from a simple definition: an indicator is a measure tracked over time and linked to a specific decision. The difficulties stem mainly from a few recurring pitfalls: the absence of a measured starting point, that is, an assessment of the process as it stands before any change, the choice of indicators that flatter but serve little purpose, and the adding up of gains that overlap one another.

This article proposes a method for reducing the risk of these pitfalls. It explains how to build a baseline, that is, the quantified pre-project reference, before any deployment. It shows how to choose indicators from the families of measurement (time and lead times, unit costs, quality, adoption and risk), and how to distinguish potential value, estimated before the project, from realised value, observed afterwards. It shows how to report results to the committee without over-interpreting them, that is, without making them say more than they actually say, and how to avoid double counting, which consists of claiming the same gain several times. Finally, it explains why the quantified objective should only be set after the diagnostic. No target value appears in this text, deliberately: every organisation has its own processes, constraints and teams, and a target borrowed from elsewhere says nothing about the actual situation.

Why the value of an AI project is hard to establish#

A conventional IT project often has a value that is easy to isolate: a data entry step eliminated, a shorter processing time, a workstation that no longer needs replacing. A generative AI project, that is, a tool that produces text, summaries, classifications or extractions from instructions, instead changes the way intellectual work is carried out. Value then depends on several interacting factors: the actual quality of the outputs, the way people take ownership of the tool, and the transformation of working rules that accompanies the change.

A gain spread across many small tasks#

When an employee saves a little time on each of their repetitive tasks, nobody notices it individually. Added up across an entire team, this gain may represent a share of the work that must be measured rather than assumed. The trap is twofold: the gain is too small to be perceived and too scattered to be measured without a method. It is therefore necessary to define a unit of work, that is, the stable object being counted, such as a file, an invoice, a page of a report or a response to a call for tenders, so that what happens before and after can be compared.

The demonstration trap#

A demonstration is designed to succeed. It relies on hand-picked cases, clean documents and carefully worded instructions. Real-world production is different: incomplete documents, atypical cases, ambiguous instructions, deadline pressure. An impressive result in the meeting room says nothing about the consistency of outputs over an ordinary working period. Measurement must therefore focus on real-world production, not on the demonstration.

What the committee really expects#

An executive committee, a board of directors or an investment committee is not looking for a brilliant statistic. It has to decide: continue, extend, adjust or stop. It wants to know whether the investment compares favourably with other options, whether the results are reliable and whether the risks remain under control. A useful measure is therefore one that informs a decision. Before choosing a single indicator, ask the following question: what decision should this indicator make it possible to take?

The measured starting point and the baseline#

These concepts underpin everything else. The measured starting point is the state of a process before any change, observed using precise, documented methods. The baseline, a term widely used in English-speaking practice, is the reference against which results will be compared: the value of the indicators before the project, frozen at a known date. Without a baseline, there is no comparison, only impressions. And an impression, even one shared by the whole team, is not enough to justify an investment.

Defining the scope#

The scope specifies which processes, which teams, which internal or external clients and which types of case fall within the measurement. A scope that is too broad dilutes the signal, and one that is too narrow makes any generalisation impossible. It is sound practice to start with a specific process whose input and output can be identified. "Preparing summary notes for the investment committee" is a workable scope. "Improving the work of the finance department" is not.

Defining the unit of work#

A unit of work must be stable, observable and comparable from one period to the next. It is defined by its entry point, that is, the moment it begins, and by its exit point, the moment it is complete and usable. For an accounting team, the unit might be a closing file or a bank reconciliation. For a due diligence team, it might be one section of a questionnaire. If the unit changes along the way, comparisons become false, and this is one of the most frequent errors.

Choosing the data sources#

The baseline relies on sources: time-stamped logs from a ticketing tool, traces from business software, tracking sheets kept by the teams, samples reviewed by hand. Each source has its blind spots. An automatic timestamp is reliable for what it records, but it ignores work done outside the system. A manually kept tracking sheet is rich in context, but it depends on the discipline of whoever fills it in. Good practice is to cross-check several sources and to document what each one does not see.

Freezing and dating the baseline#

Once the baseline has been established, it must be frozen: start date of the measurement, calculation rules, indicator definitions, people responsible for data collection. Any later change must be recorded and justified. Reconstructing a baseline after the fact is risky, for several reasons:

• Memory is selective. Teams remember the difficult episodes and forget the ordinary days, which skews the reference.
• The volume of activity may have changed. Comparing two periods with very different workloads amounts to comparing two situations that have nothing in common.
• Practices have already evolved. Sometimes in reaction to the project itself, which distorts the reference before the trial has even begun.

The four families of indicators#

An indicator only makes sense when it is linked to a decision and to a written calculation method. For an AI project, four families cover the essentials: time and lead times, unit costs, output quality, and adoption and risk. These families complement one another. A project can shorten a lead time while degrading quality, or increase usage while exposing the organisation to a new risk. Measuring them together avoids partially true conclusions, which are often the most misleading.

Time and lead times#

Cycle time is the time elapsed between a unit of work entering the process and leaving it, ready to be used. It is distinct from effort time, that is, the time a person actually spends on the unit. A file may require very little effort and still sit in a queue for a long time. Cycle time captures this waiting, whereas effort time does not. Both are useful, but they answer different questions.

The method is as follows. For each unit, record the entry date and the exit date, calculate the difference, then take the median, that is, the value below which the shorter cases lie and above which the longer cases lie. The average would be pulled upwards by a few exceptional cases. It is also necessary to look at the distribution, that is, the complete shape of the durations, because two processes with the same median can have very different extreme cases. Finally, break the cycle time down step by step to identify where time is being lost.

• Effort time measures the actual workload. It is useful for assessing the capacity freed up, but it ignores the waiting borne by the internal client or the decision-maker.
• Waiting time can account for the largest part of the cycle time. A tool that speeds up one step has no effect if the file then waits for an approval that, for its part, has not changed.
• Rework time is the time spent correcting an output before it can be used. It must be counted in the cycle time; otherwise the apparent gain is artificial.

Freed-up capacity#

Freed-up capacity is the volume of work the organisation can devote to other activities thanks to the project. It is calculated from the time saved per unit, multiplied by the number of units processed over the period, then multiplied by the share of that time actually reallocated to a useful activity. This last factor is decisive. Time saved that dissolves into less important tasks, or into additional meetings, does not create freed-up capacity in the economic sense. The measurement must therefore be accompanied by tracking, however simple, of how time is allocated.

Unit costs#

The unit processing cost is the total cost of a process over a period, divided by the number of units processed over that same period. It is useful because it brings heterogeneous expenses down to a single, comparable reference. To be reliable, the total cost must include fully loaded staff time, that is, the full cost of labour, including employer charges, relative to the hours actually worked. It must also include licences, infrastructure, supervision, training and rework. Some projects count only licences and thus show a misleadingly low unit cost. The consistency rule is simple: the period of the total cost and the period of the volume must be identical.

• Direct costs comprise the time of staff assigned to the process, licences and infrastructure. They are the easiest to record, but they are not enough.
• Supervision and training costs correspond to the time spent overseeing usage, training new joiners and updating instructions. They appear after launch and are often overlooked.
• Rework costs are the additional work required to correct, complete or check outputs. They can cancel out part of the productivity gain if they are not measured.

Output quality#

Output quality measures how far the result produced conforms to the expected criteria. It is defined in advance by means of a grid: completeness, factual accuracy, presence of sources, compliance with internal rules, tone and format. Each criterion is assessed on a simple scale, for example compliant or non-compliant, or by levels of severity if not all errors are equal. The reference method is to draw a random sample of outputs and have it assessed by a person who did not take part in producing them. The resulting indicators are the proportion of outputs accepted without modification, the breakdown of corrections by type and the severity of the errors found.

A manual process is not perfect either. The useful question is therefore not whether quality reaches some abstract ideal, but whether it is at least equivalent to the baseline, and in which types of case it deteriorates.

Adoption and risk#

The adoption rate is the share of eligible people who actually use the tool over a given period, according to a usage criterion defined in advance. Its calculation is straightforward: the number of active users under that criterion, divided by the number of eligible users. The trap lies in the definition of the word "active". A simple login proves nothing. Useful usage is measured by concrete actions: an approved output, a document produced and passed on, a real case processed.

The risk indicator measures the organisation's exposure to an identified risk: outputs used without verification, documents containing sensitive information submitted to an unauthorised tool, unsourced claims carried over into a deliverable, excessive dependence on a single provider. It is calculated by counting the events reported or detected, relative to the volume of activity. This ratio makes it possible to check that a risk does not appear low simply because activity has fallen.

The alert indicator is a signal that triggers a predefined action when it crosses a threshold set in advance. It is distinct from the risk indicator: the risk indicator measures exposure, while the alert indicator triggers the response. The threshold must be decided by the project's governance, not by the team running the project. It must be set after the diagnostic, on the basis of the baseline. A threshold without an associated procedure is nothing more than a colour on a dashboard.

Summary table of indicators#

Potential value and realised value#

These concepts must be rigorously distinguished, because confusing them is probably the costliest mistake in an investment case.

Potential value is an estimate of the contribution a project can make, built from the diagnostic and from explicit assumptions, before or during deployment. It answers the following question: if the tool were used as planned, by everyone concerned and on all eligible cases, what would the effect be?

Realised value is the contribution actually observed after deployment, measured against the baseline, using the same calculation method as in the diagnostic. It answers a different question: what actually happened in the organisation?

Why the two are never the same#

Potential value is an assumption, whereas realised value is an observation. The following table summarises the differences.

Analysing the gap#

Most often, realised value turns out to be lower than potential value. This gap is not a failure in itself: it is information. It breaks down into identifiable causes: lower adoption than expected, poorer quality on certain types of case, underestimated rework time, an unchanged downstream process that absorbs the gain. An analysis of the gap that isolates these causes is far more valuable than a single final figure, because it shows what needs to be corrected.

Presenting both without mixing them#

In any document intended for the committee, the two values must appear on separate lines, with explicit labels. The assumptions behind the potential value must be listed with their source. A presentation that displays a single amount, without qualification, invites the reader to treat it as an achieved result, when it is only an estimate.

Turning time saved into value without inventing it#

Putting a monetary value on time is a tempting step, because it allows a project to be compared with other investments. It is also the step most prone to drift. The principle is simple: first measure physical quantities (time, volume, quality, usage rates), then apply a separate valuation whose origin is stated.

Theoretical time and effective time#

Theoretical time is the time the tool is supposed to save, according to tests or the demonstration. Effective time is the time observed in production, once adoption, rework and waiting have been taken into account. It is generally lower than the former. Valuing theoretical time amounts to valuing a situation that does not yet exist.

The fully loaded hourly cost#

To value time, the fully loaded hourly cost is generally used, that is, the full cost of a person's labour, including employer charges, relative to their effective hours. It should be set by the finance department according to a stable rule, so that it does not vary from one case to another. A valuation built by the project team, without external validation, risks losing its credibility with the committee.

What valuation does not prove#

Valued time is not a saving. A saving presupposes that an expense disappears: a position not refilled, a service not ordered, an expense avoided. Freed-up capacity only becomes a saving if it translates into a concrete decision. That is why reports must distinguish capacity gains, cost savings and effects on revenue or service quality, and never add them together without due care.

Human review and sampling: measuring without marking your own homework#

Uses of human review#

Human review refers to the checking of an output by a competent person. It serves very different purposes. In day-to-day production, it is a control: no unchecked output should go out to a client or reach a committee. For measurement, it becomes an assessment instrument. The trap is to confuse these two roles. If the reviewer corrects the output before scoring it, the quality indicator actually measures the reviewer's ability to repair, not the tool's. Control review, which corrects, must therefore be kept separate from measurement review, which simply scores against the grid, on a separately constituted sample.

Sampling#

Sampling consists of selecting a subset of outputs in order to estimate the properties of the whole. It is indispensable, because reviewing every output is generally not realistic. A few principles should guide the choice:

• Random selection. Each unit has the same chance of being selected. This is the only way to prevent the person building the sample from picking, consciously or not, the most favourable cases.
• Stratification. The population is divided into homogeneous groups, for example by type of case or by level of complexity, and a sample is drawn from each of them. Without this, a rare type of case can disappear from the measurement.
• An appropriate size. The sample size depends on the variability of the phenomenon being measured and on the precision sought. The more heterogeneous the cases, the more of them need to be reviewed. A result based on a small number of cases must be presented with its limitations.

The most common danger is reviewing mainly the easy cases, because they are quick to process. Quality then appears excellent, when it is only excellent on the simple cases.

Comparing before and after, with a control group#

The before-and-after comparison sets the indicators measured before deployment against those measured afterwards. It is simple to set up and often sufficient for initial monitoring. However, it attributes to the tool everything that changes between the two points in time, including things that have nothing to do with it.

The control group is a set of comparable people or cases that continue to work without the tool, while the treated group uses it. The difference between the two groups gives a more robust estimate of the project's own effect, because it neutralises part of the changes common to both.

A bias is a systematic distortion that skews the measurement in a given direction. The simple comparison is subject to several:

• Seasonality, that is, variations linked to the calendar. Closing, reporting or year-end periods change the volumes and the nature of the work. Comparing two different times of the year amounts to comparing two different activities.
• The novelty effect. At the start of a deployment, teams are more attentive, more motivated and more willing to experiment. Performance may then fall, or settle at a level different from that of the early days.
• The observer effect. A person who knows they are being measured changes the way they work. The measurement itself changes the process being observed.
• Simultaneous changes. A new organisational structure, recruitment, a new business tool, changes in clients or in regulation: each can explain part of the observed change.

Building a control group#

Several methods exist. A staggered roll-out introduces the tool in successive waves: the teams that have not yet received it serve as a reference for those already using it. Assignment by random draw, where acceptable, is in principle the most robust, because it tends to make the groups comparable on both known and unknown characteristics. For a small project, an informal control group remains useful: one team continues with the existing method for a period set in advance. In all cases, the control group must be defined before launch, not chosen after the fact to suit the result.

Avoiding double counting#

Double counting consists of attributing the same gain to several indicators, several projects or several lines of a value statement. It is a frequent problem, because the gains of an AI project are interlinked: the same task can be accelerated, require less rework and be better documented all at once. Each of these improvements is real, but they do not all add up.

Typical situations#

A few situations come up time and again:

• A reduced processing time is counted once in time saved, a second time in unit cost savings, and yet again in freed-up capacity. Yet it is only a single gain, seen from several angles.
• A second automation, added to an already automated process, claims the entire reduction in lead time, whereas the overall gain can only be attributed once.
• An improvement in quality is counted as a rework saving, even though it is already reflected in the unit processing cost, which specifically includes rework.

A control method#

Several rules help to keep the problem under control:

• Build a value tree. Start from the total expected gain and break it down into elementary gains. Each elementary gain has only one parent, that is, a single higher level to which it is attached.
• Choose a single unit of account. This is the common quantity in which all gains are expressed, whether hours, euros or units of output. Conversion from one unit to another must be documented.
• Attribute each gain to a single project. An ownership rule states which project claims which gain. If two projects act on the same process, the shared portion is presented separately.
• Maintain a contribution matrix. Rows represent projects, columns represent gains. Check that no gain is claimed in two places.
• Isolate combined effects. When several projects together produce an effect that none produces alone, that effect must appear on a separate line, without being allocated arbitrarily.

Frequent overlaps#

Vanity metrics#

A vanity metric is a figure that rises easily and flatters whoever presents it, but says nothing about the value created. It exists because it is simple to produce and reassuring. It becomes dangerous when it takes the place of useful indicators on dashboards.

There are many examples:

• The number of documents generated. It measures the tool's output, not the quality or usefulness of the documents. An unused document counts as much as a decisive one.
• The number of accounts created. It measures initial interest, not usage. A dormant account is a cost, not an asset.
• The number of training hours delivered. It measures training effort, not the skills acquired or their application at work.
• The number of queries submitted. It measures activity, not the relevance of the answers or whether they are reused.

A simple test helps to spot them. What decision does this indicator serve? And what would we do if its value fell sharply? If the answer is vague, the indicator is probably a vanity metric.

To replace it, look for the indicator connected by a chain of causes. The number of documents generated is replaced by the proportion of documents accepted without correction, and the number of accounts created by the adoption rate under a concrete usage criterion.

Setting the quantified objective after the diagnostic#

It is often the first question asked: what is the objective? Yet it is premature to answer it before the baseline has been established. An objective set before the diagnostic rests on intuition, sometimes on a figure heard elsewhere or on a sales presentation. If it is too ambitious, the team may be tempted to circumvent the measurement. If it is too modest, the project is never pushed to raise its game.

The sequence that works#

The recommended order is as follows: framing the decision, diagnosing the process, freezing the baseline, documenting the assumptions, estimating the potential value, defining the alert thresholds, and finally having the committee set the objective. Each step provides the next with the material it needs. The objective then becomes a decision grounded in the diagnostic, not a wish.

Outcome objectives and steering objectives#

An outcome objective relates to a value indicator, for example the unit processing cost. A steering objective relates to a necessary condition, for example the level of effective user training or compliance with a review grid. Both must be set, but the second is often more useful at the outset, because it prepares the ground for the first. An outcome objective set too early, without a stable measurement process, can only produce poor information.

Setting a target without freezing it#

A useful target is expressed as a relationship with the baseline, accompanied by a revision rule. The committee needs to know what justifies the target and when it will be re-examined, particularly if the assumptions about adoption or quality prove wrong. The figure itself is a matter for governance, after the diagnostic, not for the team running the project.

Reporting to the committee without over-interpreting#

Over-interpretation consists of drawing conclusions from a measurement that it cannot support. It is very tempting, because project teams need to show that their work has borne fruit, and committees expect clear-cut conclusions. A good report holds clarity and caution together.

The structure of an honest report#

A report to the committee benefits from following a simple order:

• The decision question. What decision must the committee take, and on what basis?
• The scope and the period. What was measured, over what period and on which units of work.
• The method. Sources, definitions, sample, any control group, calculation rules.
• The results. Indicators from each family, presented with their distribution and their uncertainty interval, that is, the range of plausible values around the measurement, which reflects what the sample allows us to assert about the whole.
• The limitations. What the measurement does not allow us to assert, possible biases, cases not covered.
• The proposed decisions. Continue, extend, adjust or stop, with the associated conditions.

The most common presentation errors#

• Presenting an average on its own. An average often conceals very different cases. Show the median and the distribution.
• Confusing correlation with causation. A correlation is a simple concordance between two quantities or two events. Causation implies that a cause actually produces its effect. An improvement observed after deployment is not necessarily due to the deployment. Without a control group, this must be stated.
• Presenting potential value as realised. The difference is fundamental, and committee members will remember it.
• Generalising from a partial scope. A result obtained in one team does not hold for the whole organisation.
• Changing the definition along the way. If an indicator is modified, explain why and recalculate the previous series.
• Leaving out the failures. Poorly handled cases are the most valuable source of lessons.

Writing without adjectives#

A sentence such as "the project has transformed the way the team works" cannot be verified. A sentence such as "on the reviewed sample, the proportion of outputs accepted without correction improved relative to the baseline, with a wide uncertainty interval on complex cases" can be verified. The tone should remain sober. Adjectives, whether enthusiastic or alarmist, almost always signal a conclusion that goes beyond the data.

Governance, risks and regulatory framework#

Measuring value cannot be separated from measuring risk. A project that increases productivity while exposing the company to undetected errors has not created net value. The risk and alert indicators presented above serve precisely to establish this complete picture.

On the regulatory side, the EU AI Act provides a useful frame of reference, whose content and application timetable should be confirmed with legal counsel. Its precise obligations must be checked with legal counsel, and the classification of each use case must be documented. Several principles emerge from it, which translate directly into measurement:

• The risk-based approach. Obligations depend on the use and its potential impact. An internal drafting tool generally carries less regulatory weight than a tool that influences a decision concerning an individual, to be checked case by case with legal counsel.
• Transparency. The people concerned may need to be informed that they are interacting with an AI system, or that content has been produced or modified by such a system, in the cases provided for by the text, to be confirmed with legal counsel.
• Human oversight. Where the text requires it, a competent person should be able to understand the output, check it and, if necessary, correct or discard it. The human review described above is a practical application of this.
• Traceability. Depending on the risk level of the use, the organisation may need to reconstruct what happened, a point to be confirmed with legal counsel: data used, tool and version, human decision taken. Without traceability, audit and measurement become difficult to carry out.

In practice, traceability feeds the baseline, human oversight structures the review, transparency is reflected in adoption, and the risk-based approach guides the choice of risk indicators.

Pitfalls to avoid#

A step-by-step method#

Frame the decision#

Start with the question the committee will have to settle, then set the scope, the unit of work and the sources. A measurement that answers no question serves no purpose, however rigorous it may be.

Build the starting point#

Measure the process before the change, freeze the definitions, date the baseline and document the sources and their blind spots. This step is the least spectacular, yet it is the one on which the credibility of everything else depends.

Choose indicators by family#

Select at least one indicator in each of the four families: time and lead times, unit costs, quality, and adoption and risk. That way, gains observed on one dimension do not mask losses incurred on another.

Estimate the potential value#

Write down the assumptions, their sources and their uncertainties, then present an estimate with its scope. Overlapping gains must be removed before the final addition.

Measure the realised value#

Apply the same method in production, on a random or stratified sample, with a human review separate from the control review, and with a control group where possible.

Review, compare, decide#

Analyse the gap between potential value and realised value, set out the limitations of the measurement, propose a decision, then set the alert thresholds and steering objectives if needed.

Glossary#

• Measured starting point: the state of a process before any change, observed using a documented method and frozen at a known date.
• Baseline: the reference value of the indicators before the project, against which later results are compared.
• Unit of work: a stable object that is counted, with a clearly defined entry point and exit point.
• Cycle time: the time between a unit of work entering a process and leaving it, ready to be used. It includes waiting.
• Effort time: the time a person actually spends on a unit of work, excluding waiting.
• Rework time: the time spent correcting or completing an output before it can be used.
• Median: the value that divides a series into two parts, one of lower values and the other of higher values. It is more robust than the average in the face of extreme cases.
• Distribution: the complete shape of a series of measurements, showing the frequency of each level and not just an average value.
• Freed-up capacity: the volume of work the organisation can devote to other activities thanks to the project, provided that this time is actually reallocated.
• Unit processing cost: the total cost of a process over a period, divided by the number of units processed over the same period.
• Fully loaded hourly cost: the full cost of a person's labour, including employer charges, relative to their effective hours.
• Output quality: the degree to which a result, produced by a tool or by a person, conforms to criteria defined in advance.
• Generative AI: a family of tools capable of producing text, images or structured data from instructions.
• Language model: software trained on large quantities of text, capable of producing, summarising or classifying text from written instructions.
• Adoption rate: the share of eligible people who actually use a tool, according to a usage criterion defined in advance.
• Risk indicator: a measure of the organisation's exposure to an identified risk, relative to the volume of activity.
• Alert indicator: a signal that triggers a planned action when it exceeds a threshold set in advance by governance.
• Double counting: attributing the same gain to several indicators, projects or value lines.
• Value tree: a breakdown of the total expected gain into elementary gains, each attached to a single higher level.
• Unit of account: the common quantity in which all gains are expressed, such as hours, euros or units of output.
• Contribution matrix: a table whose rows are projects and whose columns are gains, used to check that no gain is claimed twice.
• Vanity metric: an indicator that is easy to produce and flattering, but says nothing about the value created.
• Control group: a set of comparable people or cases that continue to work without the tool and serve as a reference.
• Staggered roll-out: the gradual introduction of a tool in successive waves, making it possible to compare teams already equipped with those not yet equipped.
• Before-and-after comparison: a comparison of indicators measured before and after a change. It is simple, but exposed to many contextual biases.
• Bias: a systematic distortion that skews a measurement in a given direction, without this being visible in any individual data point taken in isolation.
• Seasonality: variations in activity linked to the calendar, such as closing or year-end periods.
• Correlation: a statistical concordance between two quantities, which does not on its own prove a cause-and-effect relationship.
• Causation: a relationship in which a cause actually produces an effect.
• Sampling: selecting a subset of units in order to estimate the properties of the whole.
• Stratification: dividing the population into homogeneous groups, from each of which a sample is drawn.
• Human review: the checking of an output by a competent person, which can serve as a control in production or as a measurement instrument.
• Potential value: an estimate of the contribution a project can make, based on the diagnostic and on explicit assumptions.
• Realised value: the contribution actually observed after deployment, by comparison with the baseline.
• Uncertainty interval: the range of plausible values around a measurement, which reflects what the sample allows us to assert about the whole.
• Over-interpretation: a conclusion drawn from data that cannot support it.
• Risk-based approach: the principle that regulatory obligations depend on the use and its potential impact.

Printable checklist#

• The decision question is written down and approved by the committee
• The scope of the process is delimited, with an identifiable input and output
• The unit of work is defined and remains stable over time
• The data sources are listed, with their blind spots
• The baseline is measured, dated and frozen
• At least one indicator is selected in each of the four families
• The unit cost includes fully loaded staff, licences, supervision, training and rework
• A concrete usage criterion is defined for the adoption rate
• The quality grid is drafted and shared before the first measurements
• The sample is drawn at random or stratified
• The measurement reviewer is separate from the control reviewer
• The control group is defined before launch, or its limitations are explained
• The assumptions behind the potential value are listed with their sources
• The value tree is established and the gain ownership rule is applied
• The contribution matrix is checked, with no gain claimed in two places
• Vanity metrics are identified and replaced
• Alert thresholds are set by governance, after the diagnostic
• The quantified objective is set by the committee, after the baseline
• Potential value and realised value are presented separately
• Uncertainty intervals and limitations are included in the report
• Failures are documented and presented
• Regulatory verification is entrusted to legal counsel for sensitive use cases

Frequently asked questions#

Should you wait until the end of the project to start measuring?#

No. The baseline must be established before deployment, and measurement must begin with the very first uses. Waiting until the end of the project means losing the reference and makes it impossible to distinguish what is due to the tool from what is due to everything else.

Can the value of a project be measured without sophisticated analytical tools?#

Yes. A well-structured spreadsheet, a shared review grid and a randomly drawn sample can be enough for initial monitoring, within documented limits. Rigour lies in the definition of the indicators and the discipline of data collection, not in the sophistication of the analytical tools.

How long should the measurement period be?#

The period must be long enough to cover normal variations in activity, including seasonal peaks, and short enough for the measurement to remain relevant. It is set before launch, according to the cycle of the process under study, and is not changed along the way to suit the result.

Is a control group essential for a small project?#

No, but its absence must be stated. Without a control group, a before-and-after comparison remains possible, provided its limitations are set out: seasonality, novelty effect, simultaneous changes. An informal control group, even an imperfect one, already helps to put the results in better context.

What should you do if realised value is lower than potential value?#

You should neither jump to the conclusion that the project has failed, nor hide the gap. Its causes need to be analysed: adoption, quality on certain types of case, rework, an unchanged downstream process. This analysis guides the corrections and allows the committee to decide with full knowledge of the facts: continue, adjust or stop.

Conclusion#

Measuring the value of an artificial intelligence project is not a technical exercise reserved for specialists. It is a decision-making discipline. It starts with a precise question, draws on a measured starting point and a frozen baseline, relies on four complementary families of indicators, and guards against vanity metrics, double counting and over-interpretation. It distinguishes potential value, which is an assumption, from realised value, which is an observation, and it sets the quantified objective only once the diagnostic is complete.

The common thread is simple: a measurement is only worth something if it enables a decision, and a decision is only sound if it rests on a method known to all. The methods presented here promise no particular result. What they do make possible is knowing what we know, what we do not yet know, and what would need to be measured to find out. For use cases involving decisions about individuals, sensitive data or regulatory obligations, verification with legal counsel remains essential.