Auditing Loyalty Data With AI: Most of the Job Is Not AI

The short version: duplicate customer identities are a defect worth auditing early, because they distort segments quietly and the damage compounds. This is the shape of that job — not a runbook. The model comes last: normalisation and deterministic matching resolve whatever they can, blocking is what makes the problem computable at all, and the model judges only the ambiguous remainder. The step most write-ups omit is measuring the result against a slice you checked exhaustively, and that is the step that decides whether any of it worked.

Bad loyalty data does not announce itself

Some data problems fail loudly: a broken export, a schema change, a job that errors. Those get fixed, because something is visibly wrong.

Identity fragmentation fails quietly. You still get five segments. Champions still have the highest scores. The report renders. Nothing tells you that your best customer exists as three profiles — a work email, a personal email, and a guest checkout — each looking like an ordinary two-order buyer, none looking like the ten-order customer they actually are.

The consequence is not a wrong chart. It is a reward budget aimed at people who were never going to leave, while the fragmented customer sits in a segment that gets a win-back discount they did not need.

The RFM spreadsheet method recommends inspecting twenty records before scoring: the five highest monetary values, five highest frequencies, five longest recencies, and five at random. That is a sound smoke test and it is what it says it is — a check for gross errors before you trust the file. It cannot tell you how many duplicates are in eighty thousand profiles, and it does not claim to.

Start with normalisation, not with a model

Before anything clever, flatten the obvious variation:

• Emails: lowercase, strip dots and +tags where the provider ignores them, trim whitespace.
• Phones: strip formatting, normalise to E.164, handle the local-vs-international prefix for your markets.
• Names: case-fold, strip titles and punctuation, normalise accents.
• Addresses: standardise to a postal format if you have a library for your country.

Then match deterministically on the identifiers that are supposed to be unique: normalised email, normalised phone, payment token, loyalty card number. Where two profiles agree on one of those, you usually have a duplicate and you did not need a model to say so.

How much this catches depends entirely on your data — how many customers use one email across purchases, whether guest checkout captures a phone, whether your payment provider exposes a stable token. Measure it on your own file rather than trusting anyone's figure, including this one. The point is that this step is cheap and its output is auditable, so it should absorb everything it can before you spend money on inference.

Blocking: why you cannot just ask the model

Here is the constraint that decides the whole design. Eighty thousand profiles produce roughly 3.2 billion possible pairs. You cannot send that to anything. Even at a hundredth of a cent per comparison it is an unaffordable job, and most of those pairs are two people who share nothing.

So you generate candidates first. Group profiles into blocks that share something cheap and discriminating, and only compare within blocks:

• Same phone suffix (last six digits)
• Same first three characters of surname plus year of first order
• Same normalised street number plus postcode outward code

Choose keys that are discriminating. A key that groups everyone in a dense urban postcode produces a block of thousands and has not reduced anything — check the size distribution of your blocks before trusting a key, and drop any that produce a long tail of huge ones.

Use several keys, not one: a single key misses every duplicate where that field is wrong or absent, which is exactly the population you are hunting. Their union is what shrinks the problem, and by how much depends on your data — measure it.

This step is pure code. No model. It is also the step most "use AI to dedupe your CRM" advice omits, which is how you end up with a workflow that cannot run on real data.

Where the model actually earns its place

You now have a few thousand candidate pairs. Compute comparison features for each one — string distance on the name, whether postcodes match, days between first orders, overlap in purchased categories, whether the domains differ — and let a scoring step rank them.

For clearly-similar and clearly-different pairs, a threshold on those features is enough. The model is for the middle: two records that share a surname and a city but differ in everything else, a business name against a personal name at the same address, a transliterated name spelled two ways.

Two rules that matter more than the model choice:

Explanations must come from the computed features, not from the model's own account of itself. "Same postcode, card last four matches, names differ by one edit" is checkable. A free-text rationale the model composed can describe evidence that is not there.

The output is a queue, never an action. Nothing merges automatically. This is not general caution — a wrong merge is unusually hard to reverse, because the record you would need to undo it is the one that got absorbed.

Measure it against a slice you checked completely

This is the step that gets skipped, and skipping it is how you end up confident and wrong.

The intuitive check is to sample: take twenty pairs the model flagged and twenty records it passed, and inspect them. The first half is fine — it estimates precision, roughly, on a small sample. The second half does not work, and it is worth seeing why.

Suppose duplicates are 2% of your records and the tool misses half of them. Then about 1% of the records it passed are misses. Sample twenty of them and your chance of catching even one is about 18%. You will almost certainly see nothing, and "I checked and found no misses" is precisely the wrong conclusion to draw from a test that fails to fire four times out of five.

Do this instead. Take a narrow slice you can examine exhaustively and find every duplicate in it by hand. That gives you a labelled set with a known denominator, which is what the sample lacked. Run the tool over the same slice and compare: what fraction of real duplicates it found, and what fraction of its claims were right.

Define the slice carefully, because the obvious choices leak. A surname slice misses the duplicate who married and changed name; a one-month order slice misses the same person's orders in other months. Search each in-slice profile against the whole customer table, not just against the slice, and write down the rule you used to call something a match before you start — otherwise you are labelling to match what the tool found.

A small complete slice beats a large random sample, because it is the denominator that makes the numbers mean anything. This also fixes the reverse error: precision of 45% sounds poor, but if duplicates are 1% of pairs it means the tool concentrated the problem forty-five-fold, and that may be an excellent queue to work through. Precision without a base rate is not interpretable in either direction.

What you send, and what you keep

The article you are reading recommends sending customer records to a model. Say plainly what that means: unless you are running something locally, those records leave your infrastructure and land with a third party under their retention and training terms.

Minimum discipline: send the computed comparison features rather than raw records wherever the judgement allows it; drop every field the decision does not need; check whether your provider trains on submitted data and turn that off; confirm the arrangement is covered by your processor agreements before anything is exported, and that your retention and deletion terms cover it.

Two traps worth naming. Hashing is not anonymisation — an email or phone hash is pseudonymous and trivially reversible by enumeration, so a hashed identifier still carries the obligations of the original. And payment tokens are payment data: if you use a token prefix as a blocking key, that key stays inside your own infrastructure, and only the resulting comparison feature — matched or not — travels.

This is a summary, not a compliance assessment. A deduplication project that creates a disclosure problem has not improved your data.

Two jobs where AI is the wrong tool

Worth stating, because both get sold as AI use cases.

Points ledger reconciliation is accounting, not inference. Whether an account's balance rolls forward correctly, whether an adjustment carries a reason code, whether an approval exists — these are deterministic checks with exact answers, and a model can only make them less certain. Note also that individual earn and burn events are not supposed to balance, and negative balances are sometimes legitimate after a reversal, so the rules must encode your actual program terms. See points liability controls for the framework this sits inside. A model is useful at one edge only: classifying free-text adjustment notes after the deterministic checks have run, and even then with a defined taxonomy and an "unclear" option.

Segment drift is a profiling job. If Champions shrank this month, compare field distributions, null rates, and volumes against previous periods, and check your deployment log. That is measurement. A model asked to explain the shift will produce a plausible narrative, which is worse than no answer because it is persuasive.

What to run this month

1. Normalise email, phone and name on a copy of your customer table. Count how many exact duplicates appear on each identifier alone. This costs an afternoon and may be most of your answer.
2. Build two or three blocking keys and count the candidate pairs they produce. If the number is not in the thousands, adjust the keys before going further.
3. Pick a slice you can check exhaustively and label it by hand. Do this before running anything, so the tool cannot influence what you count as a duplicate.
4. Score the candidates, measure against the labelled slice, and only then decide whether the queue is worth working.
5. After merging confirmed duplicates, recompute your retention inputs from scratch — frequency, contribution margin, cohort retention — rather than adjusting the old figures. The retention calculator recomputes lifetime value from those inputs, so feed it the corrected ones and compare. Merging generally raises per-customer frequency, but the direction of the final number depends on which inputs moved. Keep the margin-based definition the whole way through.

Key takeaways

• Identity fragmentation is quiet and survives into the budget, which is what makes it worth auditing early.
• Normalisation and deterministic matching come before any model. How much they resolve depends on your data — measure it before buying anything.
• Blocking is what makes deduplication computable — eighty thousand profiles are 3.2 billion pairs.
• Judge the result against a slice you checked exhaustively. Sampling the records a tool passed cannot measure what it missed.
• Precision means nothing without a base rate; 45% may be excellent or useless depending on the denominator.
• Keep it read-only, send the minimum, and confirm your privacy policy already covers it.