Levenshtein distance, Jaro-Winkler, and token similarity all measure text similarity, but they reward different kinds of resemblance. This guide explains how each fuzzy matching algorithm works, where it fails, and how to choose or combine metrics for names, addresses, product titles, and entity resolution.
Overview
Approximate string matching is useful when two values may refer to the same thing but are not identical. A user might search for “Jon Smyth” instead of “John Smith”, a customer record may contain “10 High St” while another uses “10 High Street”, or a product feed may contain punctuation, reordered words, and inconsistent model numbers.
No single similarity score is best for every task. Levenshtein distance focuses on the number of character edits needed to transform one string into another. Jaro-Winkler is designed to reward character agreement, particularly at the beginning of short strings. Token similarity compares words or fragments, making it more tolerant of changes in word order and extra terms.
The practical choice depends on the shape of the data, the cost of false matches, and whether the system is searching, deduplicating, or resolving records. For a broader testing approach, see the Fuzzy Matching Algorithm Benchmark. Treat its methods as a starting point, then test against examples from your own domain.
How to compare options
Compare algorithms against the errors and variations your application actually receives, rather than against invented examples. Build a small evaluation set containing positive pairs that should match and negative pairs that should not. Include ordinary variations as well as difficult cases: typos, abbreviations, transliteration, missing fields, swapped words, duplicated words, and misleadingly similar values.
Before calculating similarity, define your normalisation policy. Case folding, whitespace handling, punctuation removal, accent treatment, and standard abbreviations can change results more than the choice of metric. Normalisation should remove irrelevant formatting without erasing information that distinguishes entities. The query normalisation checklist covers the main decisions.
Evaluate more than average similarity. For entity resolution and record linkage, measure precision, recall, the number of candidate pairs requiring review, and the consequences of an incorrect match. A high score does not automatically mean two records represent the same entity. Read the entity resolution metrics guide for a framework that separates these concerns.
Also measure operational behaviour. A character-level comparison may be accurate but expensive when applied to every record in a large dataset. Candidate generation, indexing, length limits, and field-specific rules often matter as much as the final scoring function.
Feature-by-feature breakdown
Levenshtein distance
Levenshtein distance counts the minimum number of single-character insertions, deletions, and substitutions required to change one string into another. “Martha” and “Marhta”, for example, contain the same letters but differ in order; a character-edit measure will reflect that difference rather than treating the values as identical.
It is a strong baseline for typo-tolerant search, short codes, usernames, and values where character order matters. A normalised score is often more useful than the raw distance because the same number of edits has a larger effect on a short string than on a long one.
Its main limitation is that every edit is usually treated similarly. A transposition, a missing space, and a meaningful character substitution may not have the same business impact. Plain Levenshtein distance also does not understand word boundaries or synonyms. For production use, consider length-aware thresholds and separate rules for fields such as postcodes, SKUs, or identifiers.
Jaro-Winkler
Jaro-Winkler compares matching characters while allowing limited positional displacement, then gives an additional preference to a shared prefix. This makes it a useful name matching algorithm when short strings share their opening characters but contain small spelling variations.
It can perform well for personal names and other compact labels, particularly when the start of the value is informative. However, the prefix bonus is an assumption, not a universal truth. It may overvalue strings that begin alike but diverge later, and it is less naturally suited to long text, reordered address components, or product descriptions containing several independent terms. A detailed comparison of these two metrics is available in Jaro-Winkler vs Levenshtein for names and short strings.
Token similarity
Token-based methods split text into words or other units, compare the resulting sets or sequences, and combine the overlaps into a score. Depending on the implementation, token order may be ignored, repeated terms may be handled differently, and extra words may be penalised to varying degrees.
This makes token similarity a natural choice for addresses, company names, product titles, and search queries where word order, punctuation, or additional descriptors vary. “Blue cotton shirt medium” and “medium blue cotton shirt” can be considered close even though their character sequences differ substantially.
Token methods can fail when tokens are common, ambiguous, or incorrectly extracted. “Apple” and “Apple computer” may appear similar despite requiring different handling in a catalogue. Tokenisation also requires care with hyphens, model numbers, apostrophes, compound words, and multilingual text. For search, combine token scores with exact field matches, prefix logic, and domain-specific boosts rather than relying on one number.
Combining metrics
Many reliable systems use a staged approach. First normalise values and generate plausible candidates. Then calculate field-specific scores: perhaps Jaro-Winkler for a name, token similarity for an address, and Levenshtein for a reference code. Finally, combine the signals with explicit weights or decision rules.
Keep the model understandable. A rule such as “strong name similarity plus matching postcode” is easier to audit than an unexplained aggregate score. Store the component scores and the decision reason so that reviewers can investigate false positives and false negatives.
Best fit by scenario
- Names: Start with Jaro-Winkler and compare it with Levenshtein on initials, swapped names, diacritics, and transliteration. Add exact agreement on other fields before merging records.
- Addresses: Normalise abbreviations and split components where possible. Use token similarity for street and locality text, while comparing postcodes or house numbers with stricter rules.
- Product titles: Use token similarity for word order and extra descriptors, then protect brand, size, colour, and model fields from being treated as interchangeable.
- Typo-tolerant search: Use Levenshtein-style matching for short misspellings, but restrict candidates by prefix, field, or index to protect relevance and response time. The guide to product search with fuzzy matching covers these trade-offs.
- Deduplication and entity resolution: Use multiple fields and a review band. Do not auto-merge solely because one text field exceeds a threshold; the cost of a false positive may be much higher than the cost of manual review.
- Log and error search: Prefer conservative fuzzy matching alongside exact terms, structured filters, and timestamps. Loose similarity can return unrelated messages that share common technical words.
When to revisit
Revisit the comparison whenever your input data, normalisation rules, or user behaviour changes. New suppliers may introduce different address formats, a product catalogue may add more near-duplicate titles, or an international expansion may expose weaknesses in tokenisation and character handling. Multilingual fuzzy matching deserves its own test set rather than assuming that an English-language configuration will transfer unchanged.
Set a regular review cycle for thresholds and inspect sampled matches from production. Track false positives, missed matches, latency, and the proportion of cases sent to human review. A threshold that worked on an initial dataset can become unsuitable as the distribution of names, products, or queries changes.
To act on the comparison, create a labelled sample, document normalisation, choose a baseline metric, and test alternatives against the same examples. Start with conservative thresholds, log component scores, and introduce field-specific combinations only where the baseline is insufficient. For implementation details, the guides to building a fuzzy search API and CRM data cleanup provide useful next steps.
