Customer records rarely arrive in a consistent format. Names may contain different spellings, email addresses may include old domains, phone numbers may use different country formats, and addresses may be abbreviated or reordered. This checklist presents a repeatable entity resolution workflow for finding likely duplicates, scoring matches, routing uncertain cases to review, and recording safe merge decisions without treating fuzzy similarity as proof of identity.
Overview
Entity resolution, also called record linkage or customer record matching, is the process of deciding whether two records refer to the same real-world customer. It is more demanding than a simple duplicate search because no single field can always be trusted. Two records may share an email address but have different names, or have matching names but belong to different people.
A robust deduplication workflow separates four activities:
- Preparation: preserve the original data and create consistent comparison fields.
- Candidate generation: reduce the number of record pairs that need detailed comparison.
- Scoring: combine field-level evidence using rules or a fuzzy matching algorithm.
- Decision-making: automatically accept only strong matches and send ambiguous cases for review.
Fuzzy matching is useful for spelling differences, missing punctuation, transposed characters, and variation in word order. It should not be used as an automatic licence to merge records. A high text similarity score can still describe two different people, particularly when a name is common or a household shares contact details.
Start by defining the purpose of the match. Merging duplicate customer profiles, linking an imported mailing list, consolidating support tickets, and identifying repeat purchases may require different thresholds and evidence. Write down what a safe match means before tuning the scoring logic.
For background on choosing a comparison method, see Levenshtein vs Jaro-Winkler vs Jaccard. For larger datasets, candidate generation and database-side similarity can also be important; the guide to PostgreSQL fuzzy search with pg_trgm covers related production considerations.
Checklist by scenario
Before matching any customer import
- Keep an immutable copy of the source file or source records.
- Assign every incoming record a stable import identifier.
- Record the import date, source system, and transformation version.
- Separate blank values from literal values such as “unknown” or “not provided”.
- Decide which system is authoritative for each field if records are merged.
- Define whether the process will suggest matches, create a review queue, or apply approved merges.
When normalising names
- Trim leading and trailing whitespace and collapse repeated internal spaces.
- Apply consistent case folding for comparison while retaining the original display value.
- Normalise punctuation carefully; apostrophes, hyphens, and suffixes may carry useful information.
- Consider separate comparison forms for full name, given name, family name, and initials.
- Handle titles and suffixes explicitly rather than silently deleting every short token.
- Do not assume that token order is interchangeable in every naming convention.
Name matching often benefits from Jaro-Winkler or another character-oriented measure when typographical variation is expected. Token-based similarity can be more useful when names contain reordered words. Compare methods on examples from your own data rather than selecting an algorithm because it is popular. The algorithm comparison guide provides a useful framework for that choice.
When normalising email addresses
- Trim spaces and compare using a consistent case policy.
- Validate basic structure, but do not treat a syntactically valid address as proof that it belongs to the customer.
- Compare the complete address before considering local-part or domain components separately.
- Be cautious with provider-specific transformations such as removing dots or plus-tags; apply them only when the business context justifies it.
- Keep historical addresses as evidence rather than overwriting them without a retention decision.
When normalising phone numbers
- Remove presentation characters such as spaces, brackets, and hyphens for comparison.
- Use a known country or region context where available.
- Store the normalised comparison value separately from the value shown to users.
- Distinguish missing numbers from extensions, shared household numbers, and business switchboards.
- Do not infer that a shared phone number proves two records are duplicates.
When matching addresses
- Standardise case, whitespace, common punctuation, and clearly defined abbreviations.
- Compare components such as house number, street, locality, postal code, and country where they can be extracted reliably.
- Use tokenisation for reordered address components, but preserve numeric tokens and unit identifiers.
- Allow for missing secondary address lines without treating every partial match as equivalent.
- Account for multilingual data, transliteration, and local address conventions instead of forcing every address into one language.
The multilingual matching guide on normalisation, transliteration, and name matching is relevant when the same customer data can appear in multiple scripts or language conventions.
When building candidate pairs
Comparing every record with every other record becomes expensive and can produce an overwhelming review queue. Use blocking, sometimes called candidate blocking, to create plausible pairs first. Blocks might use a postal-code prefix, email domain, phone suffix, house number, or a normalised name key. Use more than one blocking rule when a single field is incomplete or unreliable.
Blocking rules should be permissive enough to avoid losing genuine matches. Keep track of which rule produced each candidate so reviewers can understand why a pair was considered. If a record appears only because of a weak block, it may need a higher review threshold.
When calculating a match score
Calculate a separate similarity for each field, then combine the evidence. A simple weighted model might give greater weight to a verified email or phone match, moderate weight to a postal code and address, and lower weight to a common name on its own. The exact weights should reflect data quality and the consequences of a false merge.
Use at least three decision bands:
- Accept: strong evidence from independent fields and no material contradiction.
- Review: plausible match with missing, conflicting, or weak evidence.
- Reject: insufficient similarity or a clear contradiction.
Do not publish thresholds as universal values. A score from one library, normalisation pipeline, or field length distribution is not directly comparable with a score from another. Test thresholds against labelled examples and revisit them when the input data changes. A threshold calculator can help organise experiments, but the final decision should use business-labelled cases as well as numeric scores.
What to double-check
Validate the normalisation pipeline
Inspect before-and-after examples for every transformation. Check that accented characters, apostrophes, hyphens, apartment numbers, leading zeroes, and non-Latin scripts are handled intentionally. A normalisation rule that improves one country or data source may damage another. Always retain the raw value so a reviewer can compare the transformed value with the original.
Check for contradictory evidence
Similarity scores can hide important conflicts. A matching surname and address may not outweigh two unrelated email addresses. A phone number shared by several household members should not automatically merge their profiles. Add explicit contradiction rules, such as lowering confidence when two verified identifiers disagree.
Review the candidate distribution
Measure how many candidates each record receives, how many fall into each confidence band, and which blocking rules generate the most candidates. A sudden increase may indicate a broken normalisation rule, an overly broad block, or a changed source format. A sudden decrease may indicate that a blocking key is missing or being parsed incorrectly.
Use labelled examples
Create a review set containing confirmed duplicates, confirmed non-duplicates, difficult cases, and missing-data cases. Include common names, short names, business addresses, shared contact details, and records from each major source. Use this set to compare algorithms, weights, and thresholds. Keep the labels and decision reasons with the evaluation data so later changes can be explained.
Make every action auditable
A merge record should normally include the source record identifiers, the selected master record, field-level values before the merge, normalised values used for comparison, individual similarity scores, combined score, decision band, reviewer or rule identifier, timestamp, and reason for the decision. Also define how a mistaken merge can be reversed. An audit trail is useful for debugging as well as governance.
Common mistakes
- Using one universal threshold: short names, long addresses, and email addresses have different similarity behaviour. Calibrate by field and scenario.
- Matching on names alone: names are valuable signals but are often not unique enough for an irreversible merge.
- Over-normalising: deleting too much information can turn distinct records into identical comparison strings.
- Skipping blocking design: a slow all-to-all comparison may encourage rushed thresholds or incomplete processing.
- Ignoring missingness: two blank fields should not receive the same match credit as two equal, populated fields.
- Replacing source data: destructive cleaning removes the evidence needed to investigate a bad match.
- Auto-merging the review band: uncertainty is a reason to inspect a case, not a reason to round the score upward.
- Testing only easy duplicates: a system that succeeds on exact duplicates may still fail on the ambiguous records that create the greatest risk.
- Changing libraries without revalidation: a new fuzzy matching library or database function can produce different scores, token handling, and edge-case behaviour.
For implementation choices, compare the capabilities and trade-offs of RapidFuzz, FuzzyWuzzy, Fuse.js, and database options. If the workflow is used as part of a live search or import service, monitor it with relevance and regression checks rather than relying only on a one-time benchmark.
When to revisit
Revisit the workflow before seasonal planning cycles, major imports, migrations, or campaigns that change the volume and shape of customer data. Also review it whenever a source system, CRM field, parsing library, database extension, or fuzzy matching library changes. New countries, scripts, address formats, or customer segments are further reasons to revalidate normalisation and blocking rules.
Use this practical review sequence:
- Sample recent records from every active source.
- Compare raw values with their normalised forms.
- Run the labelled evaluation set and record changes in precision, recall, and review volume.
- Inspect accepted matches, rejected matches, and a sample from the review band.
- Check for new contradiction patterns, shared identifiers, and missing fields.
- Approve threshold or weight changes through a documented change process.
- Run a dry run before applying merges and preserve the proposed actions.
- Monitor post-merge corrections and feed confirmed decisions back into the labelled set.
Entity resolution works best as a controlled data-quality process rather than a single fuzzy matching query. Preserve original values, compare the right fields for the scenario, block candidates carefully, score evidence transparently, and keep uncertain cases reversible. That approach makes customer record matching safer to operate and easier to improve when the data or workflow changes.