Benchmarking RapidFuzz Scorers on Vendor Names

Picking a fuzzy scorer by eyeballing a handful of examples is how teams end up with a matcher that works in the demo and misfires in production. rapidfuzz ships several scorers — ratio, partial_ratio, token_sort_ratio, token_set_ratio, and JaroWinkler — and each one encodes a different assumption about how vendor names get mangled: reordering, truncation, token insertion, or prefix-weighted typos. This page builds a small labelled fixture of vendor-name pairs, runs all five scorers against it, sweeps thresholds to compute precision/recall/F1 per scorer, and picks an operating point backed by numbers instead of intuition. It assumes descriptors have already been through Choosing a String-Similarity Algorithm framing and cleaned up per Normalising Merchant Descriptors Before Matching.

Prerequisites

rapidfuzz installed (pip install rapidfuzz) — it implements scorers in C++ and is materially faster than the pure-Python Levenshtein approach used in Implementing Levenshtein Distance for Vendor Name Matching.
Descriptors already normalised (case-folded, punctuation stripped, legal suffixes handled) — an un-normalised fixture will bias every scorer toward artificially low scores.
A sample of real vendor names from your ledger, with at least 30–50 known-match and 30–50 known-non-match pairs.
Python 3.10+ with type hints; pydantic v2 for the fixture schema; Decimal reserved for any monetary fields carried alongside the pairs.
A structured logger configured to emit trace_id, source_hash, and match_decision on every scoring decision.

Step 1 — Build a labelled fixture

The fixture is the whole benchmark: without a truthful set of match/non-match pairs drawn from real vendor traffic, threshold numbers are meaningless. Each row pairs a ledger descriptor against a counterparty descriptor with a ground-truth label assigned by a human reviewer, not by a scorer.

python

from pydantic import BaseModel, ConfigDict
from decimal import Decimal
from datetime import datetime, timezone
from uuid import UUID
import hashlib

class LabelledPair(BaseModel):
    model_config = ConfigDict(frozen=True)

    pair_id: UUID
    ledger_descriptor: str
    counterparty_descriptor: str
    is_match: bool                 # ground truth, assigned by a human reviewer
    sample_amount: Decimal | None = None  # carried for context only, never scored
    source_hash: str

def make_pair(pair_id: UUID, left: str, right: str, is_match: bool) -> LabelledPair:
    blob = f"{left}|{right}|{is_match}".encode("utf-8")
    return LabelledPair(
        pair_id=pair_id,
        ledger_descriptor=left,
        counterparty_descriptor=right,
        is_match=is_match,
        source_hash=hashlib.sha256(blob).hexdigest(),
    )

FIXTURE: list[LabelledPair] = [
    make_pair(UUID(int=1), "acme industrial supply", "acme indl supply co", True),
    make_pair(UUID(int=2), "acme industrial supply", "acme logistics group", False),
    make_pair(UUID(int=3), "riverside freight llc", "riverside frt", True),
    make_pair(UUID(int=4), "riverside freight llc", "riverbank freight llc", False),
    make_pair(UUID(int=5), "global paper products", "products global paper", True),
    make_pair(UUID(int=6), "sterling data services", "sterling data svc intl", True),
    make_pair(UUID(int=7), "sterling data services", "starling data services", False),
]

A production fixture belongs in a versioned file, not inline in a benchmark script — stratify it by descriptor length, token count, and known confusable pairs (transpositions, abbreviations, subsidiary suffixes) so the sweep in Step 3 isn’t dominated by easy cases.

Step 2 — Score every pair with each candidate scorer

rapidfuzz.fuzz exposes ratio (edit-distance similarity on the full string), partial_ratio (best-matching substring, useful when one side is truncated), token_sort_ratio and token_set_ratio (token-order-insensitive, the latter also insensitive to extra tokens), plus rapidfuzz.distance.JaroWinkler (prefix-weighted, strong on short entity names with typos near the end).

python

from rapidfuzz import fuzz
from rapidfuzz.distance import JaroWinkler
from typing import Callable

Scorer = Callable[[str, str], float]

SCORERS: dict[str, Scorer] = {
    "ratio": fuzz.ratio,
    "partial_ratio": fuzz.partial_ratio,
    "token_sort_ratio": fuzz.token_sort_ratio,
    "token_set_ratio": fuzz.token_set_ratio,
    "jaro_winkler": lambda a, b: JaroWinkler.normalized_similarity(a, b) * 100,
}

def score_pair(pair: LabelledPair) -> dict[str, float]:
    return {
        name: fn(pair.ledger_descriptor, pair.counterparty_descriptor)
        for name, fn in SCORERS.items()
    }

scored_rows = [(pair, score_pair(pair)) for pair in FIXTURE]

Normalising JaroWinkler.normalized_similarity to a 0–100 scale keeps every scorer directly comparable in the sweep below — mixing a 0–1 scorer with 0–100 scorers is a common source of silently wrong thresholds.

Step 3 — Sweep thresholds and compute precision, recall, F1

Sweeping a fixed threshold grid per scorer, rather than trusting a single cutoff, exposes where each scorer’s precision/recall trade-off actually sits for your descriptor population.

python

from dataclasses import dataclass

@dataclass(frozen=True)
class SweepResult:
    scorer: str
    threshold: float
    precision: float
    recall: float
    f1: float

def sweep_threshold(scorer_name: str, rows: list[tuple[LabelledPair, dict[str, float]]],
                    thresholds: list[float]) -> list[SweepResult]:
    results = []
    for t in thresholds:
        tp = fp = fn = 0
        for pair, scores in rows:
            predicted = scores[scorer_name] >= t
            if predicted and pair.is_match:
                tp += 1
            elif predicted and not pair.is_match:
                fp += 1
            elif not predicted and pair.is_match:
                fn += 1
        precision = tp / (tp + fp) if (tp + fp) else 0.0
        recall = tp / (tp + fn) if (tp + fn) else 0.0
        f1 = 2 * precision * recall / (precision + recall) if (precision + recall) else 0.0
        results.append(SweepResult(scorer_name, t, precision, recall, f1))
    return results

THRESHOLD_SWEEP = [70.0, 75.0, 80.0, 85.0, 88.0, 90.0, 92.0, 95.0]

all_results = [
    r
    for name in SCORERS
    for r in sweep_threshold(name, scored_rows, THRESHOLD_SWEEP)
]
best_per_scorer = {
    name: max((r for r in all_results if r.scorer == name), key=lambda r: r.f1)
    for name in SCORERS
}

best_per_scorer gives one candidate operating point per scorer; the scorer with the highest F1 at its own best threshold — not the highest raw score on a few examples — is the one to promote.

Step 4 — Vectorise the comparison with `process.cdist`

Scoring pairs one at a time is fine for a fixture of dozens of rows; at production volume, comparing every unmatched ledger row against every open counterparty row needs batch scoring. rapidfuzz.process.cdist computes a full similarity matrix in compiled code, which is the same vectorisation pattern that makes optimizing pandas merge approaches viable at scale for exact keys.

python

from rapidfuzz import process
import numpy as np

def batch_score(left: list[str], right: list[str], scorer_name: str,
                score_cutoff: float = 0.0, workers: int = -1) -> np.ndarray:
    scorer_fn = fuzz.token_set_ratio if scorer_name != "jaro_winkler" else JaroWinkler.normalized_similarity
    matrix = process.cdist(
        left, right,
        scorer=scorer_fn,
        score_cutoff=score_cutoff,
        workers=workers,
    )
    return matrix

ledger_names = [p.ledger_descriptor for p in FIXTURE]
counterparty_names = [p.counterparty_descriptor for p in FIXTURE]
similarity_matrix = batch_score(ledger_names, counterparty_names, "token_set_ratio", score_cutoff=70.0, workers=-1)

workers=-1 spreads the matrix computation across all available cores; score_cutoff lets the C++ layer skip scoring pairs that can’t possibly clear the threshold, which is materially faster than filtering a fully-computed matrix in Python afterward.

Step 5 — Emit an audited match decision

Every chosen operating point feeds into a decision function that logs the scorer, threshold, and resulting classification so a later audit can reconstruct exactly why a pair was accepted or sent to exception handling.

python

import logging

log = logging.getLogger("matching.benchmark")

def decide_match(pair: LabelledPair, scorer_name: str, threshold: float) -> str:
    score = SCORERS[scorer_name](pair.ledger_descriptor, pair.counterparty_descriptor)
    decision = "MATCH" if score >= threshold else "NO_MATCH"
    log.info(
        "vendor.scorer.decision",
        extra={
            "trace_id": str(pair.pair_id),
            "source_hash": pair.source_hash,
            "match_decision": decision,
            "scorer": scorer_name,
            "threshold": threshold,
            "score": round(score, 2),
        },
    )
    return decision

Configuration boundary table

Parameter	Default	Valid range	Notes
`scorer`	`token_set_ratio`	enum (5 scorers)	Best F1 on labelled fixture wins, not intuition
`threshold_sweep`	`[70, 75, ..., 95]`	5-pt step, `50`–`99`	Coarser steps hide real boundary shifts
`prefix_weight`	`0.1`	`0.0`–`0.25`	`JaroWinkler` only; higher over-weights matching prefixes
`score_cutoff`	`0.0`	`0`–`100`	Passed into `process.cdist` to skip unreachable pairs
`workers`	`-1`	`-1` or `1`–`N`	`-1` uses all cores for `cdist`; pin to `1` for reproducible benchmarking

Verification and testing

The fixture doubles as a regression test: once a scorer and threshold are chosen, assert the resulting F1 never regresses below a floor as normalisation rules or the fixture itself evolve.

python

def test_token_set_ratio_meets_f1_floor():
    rows = [(pair, score_pair(pair)) for pair in FIXTURE]
    results = sweep_threshold("token_set_ratio", rows, THRESHOLD_SWEEP)
    best = max(results, key=lambda r: r.f1)
    assert best.f1 >= 0.85, f"token_set_ratio F1 dropped to {best.f1:.2f} at threshold {best.threshold}"
    assert best.precision >= 0.80

Re-run the sweep whenever the fixture grows or normalisation logic changes, and keep the fixture itself under version control — a benchmark against a fixture that silently drifted is worse than no benchmark at all.

Troubleshooting

UNNORMALISED_INPUT — every scorer reports suspiciously low scores across the board. Root cause: descriptors weren’t case-folded or legal-suffix-stripped before scoring, so ordinary variance (LLC vs L.L.C.) eats into the similarity budget. Fix: run the normalisation pipeline from Normalising Merchant Descriptors Before Matching before scoring, never after.
THRESHOLD_OVERFIT — a threshold scores well on the fixture but misfires in production. Root cause: the fixture is too small or too easy, so the sweep found a threshold tuned to noise rather than a real decision boundary. Fix: grow the fixture past 100+ pairs stratified by descriptor length and known confusable cases, and re-validate on a held-out split.
WRONG_SCORER — token_sort_ratio picked but truncated descriptors keep missing. Root cause: token_sort_ratio still compares full reordered strings, so a truncated counterparty name ("acme ind" vs "acme industrial supply") scores low. Fix: prefer partial_ratio or token_set_ratio when either side is known to be truncated.
SLOW_PAIRWISE — benchmark or production matching takes minutes on a modest fixture. Root cause: scoring pairs in a Python loop instead of using the vectorised path. Fix: switch to rapidfuzz.process.cdist with workers=-1 and a score_cutoff, as in Step 4.
SCALE_MISMATCH — jaro_winkler results look incomparable to the other scorers. Root cause: JaroWinkler.normalized_similarity returns 0–1 while fuzz.* scorers return 0–100. Fix: multiply by 100 before adding it to the same threshold sweep, as in Step 2.

Part of Choosing a String-Similarity Algorithm within Transaction Matching Algorithms & Logic.