Normalising Merchant Descriptors Before Fuzzy Matching Runs

Card and bank feeds emit merchant descriptors that were never designed to be compared to anything — they carry POS terminal codes, store numbers, embedded transaction dates, and city/state suffixes bolted on by the acquiring processor. Feeding that noise straight into Fuzzy String Matching Techniques inflates edit distance and drags token-based scorers toward false negatives, because the noise tokens dilute the signal the scorer is actually trying to measure. This page builds a deterministic normalisation pipeline that runs immediately before scoring: Unicode canonicalisation, processor noise-token removal, whitespace and reference-digit cleanup, and an alias/canonical-merchant lookup — with the original descriptor preserved for audit and every decision written to a structured log.

Prerequisites

Raw descriptors are captured verbatim from the card or bank feed before any comparison logic in Fuzzy String Matching Techniques runs.
A scorer has already been selected via Choosing a String Similarity Algorithm, so normalisation can target that scorer’s sensitivities (token order, substring weighting) instead of guessing.
Python 3.10+ with pydantic v2 installed.
A structured logger configured to accept trace_id, source_hash, and match_decision as extra fields.
A starter alias map — even a sparse dict of a few dozen known merchant prefixes — to seed the canonical-name lookup.

Step 1 — Casefold and Unicode-normalise the raw string

Descriptors arrive in inconsistent case and, from international acquirers, with composed or decomposed accented characters that break byte-level comparison even when the merchant name is identical. casefold() gives a locale-safer lowercase than .lower() for comparison purposes, and Unicode Normalisation Form NFKC collapses compatibility characters (full-width digits, ligatures) into their canonical equivalents before accents are stripped.

python

import unicodedata

def unicode_normalise(raw: str, form: str = "NFKC") -> str:
    """Casefold, Unicode-normalise, and strip accents from a raw descriptor."""
    folded = raw.strip().casefold()
    composed = unicodedata.normalize(form, folded)
    decomposed = unicodedata.normalize("NFKD", composed)
    unaccented = "".join(ch for ch in decomposed if not unicodedata.combining(ch))
    return unicodedata.normalize(form, unaccented).upper()

The final .upper() call fixes the canonical storage case so downstream comparisons in rapidfuzz (which is itself case-sensitive) never see two representations of the same merchant. Keep form configurable — some processors emit compatibility characters that only NFKC/NFKD decompose correctly, while NFC/NFD leave them intact.

Step 2 — Strip processor noise tokens

Store numbers, POS terminal markers, card-network prefixes (SQ*, TST*, PAYPAL *), and embedded city/state/date fragments are processor metadata, not merchant identity. Each is stripped with a configurable list of regular expressions rather than a single hard-coded pattern, because noise conventions vary by acquirer.

python

import re

DEFAULT_NOISE_PATTERNS = [
    r"\bPOS\b",
    r"\bSTORE\s*#?\d+\b",
    r"\b(SQ|TST|PAYPAL|VISA|MC|AMEX)\s?\*",
    r"\b[A-Z]{2}\s\d{5}(-\d{4})?\b",   # trailing state + ZIP
    r"\b\d{2}/\d{2}(/\d{2,4})?\b",     # embedded transaction date
]

def strip_noise_tokens(text: str, patterns: list[str] | None = None) -> str:
    patterns = patterns or DEFAULT_NOISE_PATTERNS
    for pattern in patterns:
        text = re.sub(pattern, " ", text)
    return text

Each substitution replaces the match with a single space rather than deleting it outright, so two adjacent tokens don’t get fused into a new, unintended word. That fusion is a common cause of the OVER_NORMALISED failure covered below.

Step 3 — Collapse whitespace and strip trailing reference digits

Once the noise tokens are gone, the string is riddled with runs of whitespace left behind by the substitutions, and it often still carries a trailing authorization or terminal reference number that survived because it wasn’t preceded by a recognised prefix. A configurable minimum length (min_len) guards against stripping a descriptor down to nothing.

python

def collapse_and_strip_digits(text: str, min_len: int = 3, strip_digits: int = 4) -> str:
    collapsed = re.sub(r"\s+", " ", text).strip()
    ref_pattern = r"\s?\d{" + str(strip_digits) + r",}$"
    de_refed = re.sub(ref_pattern, "", collapsed).strip()
    if len(de_refed) < min_len:
        raise ValueError(f"EMPTY_AFTER_STRIP: '{text}' collapsed below min_len={min_len}")
    return de_refed

strip_digits controls how long a trailing digit run has to be before it’s treated as a reference code rather than part of the merchant name — a merchant literally named “7-ELEVEN” must survive this step, which is why the threshold defaults to 4 rather than stripping any trailing digit.

Step 4 — Apply the alias/canonical-merchant map

Many descriptors, even after cleaning, still don’t match the name a human would recognise (AMZN MKTP versus AMAZON). A prefix-keyed alias map resolves these to a single canonical form. The raw descriptor is never discarded — it travels alongside the cleaned and canonical forms in a frozen record so any downstream review can see exactly what the processor originally sent.

python

from pydantic import BaseModel, ConfigDict

ALIAS_MAP: dict[str, str] = {
    "AMZN MKTP": "AMAZON",
    "WM SUPERCENTER": "WALMART",
    "SEATTLE COFFEE": "SEATTLE COFFEE CO",
}

class NormalisedDescriptor(BaseModel):
    model_config = ConfigDict(frozen=True)

    raw: str
    cleaned: str
    canonical: str
    alias_hit: bool
    source_hash: str

def apply_alias_map(cleaned: str, alias_map: dict[str, str] | None = None) -> tuple[str, bool]:
    alias_map = alias_map or ALIAS_MAP
    for prefix, canonical in alias_map.items():
        if cleaned.startswith(prefix):
            return canonical, True
    return cleaned, False

Longest-prefix-wins ordering matters once the alias map grows past a few dozen entries — a naive dict iteration can match a short, generic prefix before a more specific one. Sort alias_map keys by descending length before iterating in production, or store it as a trie if the map exceeds a few hundred entries; either fix eliminates the ALIAS_COLLISION failure mode below.

Step 5 — Assemble the pipeline and emit the audit log

The full pipeline runs the four prior steps in order and emits one structured log line per descriptor, carrying the trace_id, source_hash of the raw input, and the match_decision the alias step made. This is the record an auditor replays to prove a canonical name wasn’t silently invented after the fact.

python

import hashlib
import logging
from uuid import UUID

log = logging.getLogger("matching.normalise")

def normalise_descriptor(
    raw: str,
    trace_id: UUID,
    *,
    unicode_form: str = "NFKC",
    noise_patterns: list[str] | None = None,
    alias_map: dict[str, str] | None = None,
    strip_digits: int = 4,
    min_len: int = 3,
) -> NormalisedDescriptor:
    source_hash = hashlib.sha256(raw.encode("utf-8")).hexdigest()
    folded = unicode_normalise(raw, form=unicode_form)
    denoised = strip_noise_tokens(folded, noise_patterns)
    cleaned = collapse_and_strip_digits(denoised, min_len=min_len, strip_digits=strip_digits)
    canonical, alias_hit = apply_alias_map(cleaned, alias_map)
    decision = "ALIAS_HIT" if alias_hit else "NORMALISED_ONLY"

    log.info(
        "descriptor.normalised",
        extra={
            "trace_id": str(trace_id),
            "source_hash": source_hash,
            "match_decision": decision,
            "raw": raw,
            "canonical": canonical,
        },
    )
    return NormalisedDescriptor(
        raw=raw,
        cleaned=cleaned,
        canonical=canonical,
        alias_hit=alias_hit,
        source_hash=source_hash,
    )

Only the canonical field should ever be handed to the scorer selected in Choosing a String Similarity Algorithm — passing raw or cleaned into token_set_ratio reintroduces exactly the noise this pipeline exists to remove. The measurable effect of that discipline is quantified in Benchmarking RapidFuzz Scorers on Vendor Names, where normalised inputs consistently raise scorer separation between true and false matches.

Configuration boundary table

Parameter	Default	Valid range	Notes
`unicode_form`	`NFKC`	`NFC`\|`NFD`\|`NFKC`\|`NFKD`	Compatibility form recommended; applied before accent stripping
`noise_tokens`	`DEFAULT_NOISE_PATTERNS`	list[str] (regex)	Processor/network/date noise removed prior to comparison
`strip_digits`	`4`	`3`–`8`	Minimum trailing digit-run length treated as a reference code
`alias_map`	`{}` (seeded from canonical-merchant table)	dict[str, str]	Prefix-keyed lookup; sort by descending key length
`min_len`	`3`	`1`–`12`	Cleaned length below this raises `EMPTY_AFTER_STRIP`

Verification and testing

Pin a known-noisy fixture and assert the exact canonical output, the alias decision, and that the noise tokens are actually gone — a passing test that only checks the final string can hide a regression where an unrelated word survives the strip step.

python

def test_normalise_descriptor_fixture():
    raw = "SQ *SEATTLE COFFEE STORE #4471 SEATTLE WA 98101 06/14 003391"
    result = normalise_descriptor(raw, trace_id=UUID(int=42))

    assert result.canonical == "SEATTLE COFFEE CO"
    assert result.alias_hit is True
    assert "STORE" not in result.cleaned
    assert "98101" not in result.cleaned
    assert result.source_hash == hashlib.sha256(raw.encode("utf-8")).hexdigest()
    assert result.raw == raw  # original payload preserved for audit

Run this fixture alongside a larger regression corpus of real (anonymised) descriptors sampled from production, re-run whenever noise_tokens or alias_map changes, and diff the canonical outputs against the previous run — an unexpected shift in more than a handful of rows usually means a new regex pattern is over-matching.

Troubleshooting

OVER_NORMALISED — two distinct merchants collapse to the same cleaned string. Root cause: a noise pattern in strip_noise_tokens is too broad and eats part of the merchant name (commonly the two-letter state-code pattern matching a merchant’s initials). Fix: anchor patterns more tightly with \b boundaries and test each new pattern against the full fixture corpus before deploying it.
ALIAS_COLLISION — the wrong canonical name is applied. Root cause: apply_alias_map matched a short, generic prefix before a longer, more specific one because dict iteration order wasn’t controlled. Fix: sort alias_map keys by descending length (or migrate to a trie) so the most specific prefix always wins.
UNICODE_ARTEFACT — stray characters remain after normalisation. Root cause: unicode_form was left at NFC/NFD, which doesn’t decompose compatibility characters (full-width digits, ligatures) the way NFKC/NFKD do. Fix: set unicode_form="NFKC" unless a specific descriptor source requires strict canonical (non-compatibility) equivalence.
EMPTY_AFTER_STRIP — collapse_and_strip_digits raises ValueError. Root cause: an overly aggressive strip_digits or noise pattern removed the entire merchant name, usually on a short descriptor. Fix: raise strip_digits for that source, or exempt very short raw descriptors from the trailing-digit strip entirely.
WHITESPACE_LEAK — cleaned strings still contain double spaces reaching the scorer. Root cause: a custom noise pattern was added without routing the result back through collapse_and_strip_digits. Fix: always run whitespace collapse as the last step after any additional noise stripping, never before it.

Part of Fuzzy String Matching Techniques within Transaction Matching Algorithms & Logic.