Normalising MT940 Field 86 Remittance Info for Deterministic Counterparty Extraction

The SWIFT MT940 specification defines :86: only as “information to account owner” — a free-text bag with no mandated internal structure. In practice every bank pours its own semi-structured convention into that bag: German institutions overwhelmingly use the Deutsche Kreditwirtschaft (DK) ?NN Geschäftsvorfallcode (GVC) scheme, while many SEPA-region banks emit slash-delimited structured tags such as /EREF/, /ORDP/, /BENM/, and /REMI/ that mirror ISO 20022 remittance elements. Neither convention is announced in the file itself, and both routinely span the multi-line continuations already concatenated by the tag-aware finite state machine in How to Parse MT940 Files in Python. This page picks up exactly where that parser leaves off: given one raw, continuation-joined :86: string per transaction, detect which subfield convention produced it, tokenize the ?NN or /CODE/ segments, reassemble a hard-wrapped multi-line remittance narrative, and extract a counterparty name, IBAN, and end-to-end reference the matching cascade can actually key on. It is a narrower, deeper companion to the OFX & MT940 Parser Design reference — OFX sidesteps this entirely with a single flat <MEMO> element, which is one reason the two formats need structurally different downstream normalisation, as covered in Parsing OFX SGML Statements in Python.

Treating :86: as an opaque narrative string is the single biggest source of quiet data loss in an MT940 pipeline. A parser that only surfaces the raw text hands the matching cascade a blob it can, at best, fuzzy-match against; a parser that decomposes the subfields correctly hands it a counterparty name, an IBAN, and a machine-checkable reference — the difference between a probabilistic guess and a deterministic exact match. Because the two dominant conventions are structurally incompatible with each other (positional ?NN codes versus self-delimiting /CODE/ tags), and because a single institution can migrate from one to the other mid-contract without warning, the normalisation layer has to treat format detection as a first-class, auditable decision rather than an implementation detail buried inside a regex.

Prerequisites

• The MT940 tag-aware finite state machine described in How to Parse MT940 Files in Python has already run and multi-line :86: continuations are concatenated into one raw string per transaction.
• A per-bank subfield profile registry exists, keyed on BIC, distinguishing GVC ?NN feeds from SEPA-style slash-tag feeds — :86: carries no self-describing format marker, so this registry (not the file) is the source of truth.
• The upstream MT940Transaction or CanonicalTxn record already carries a Decimal amount field and a UTC value_date, per the OFX & MT940 Parser Design contract; this page only populates the remittance side of that record.
• Python 3.10+ with pydantic v2, re, hashlib, and logging installed.
• A structured logger configured to emit trace_id, source_hash, and match_decision on every parse attempt, success or quarantine.

Step 1 — Detect the subfield convention per bank

Because :86: is unstructured by specification, the only reliable way to know which subfield grammar produced a given line is a per-bank profile pinned by BIC. Auto-detection by regex probe is a reasonable fallback during onboarding, but it must never silently override a registered profile — a bank that happens to include a stray /XXXX/-shaped substring inside free narrative text would otherwise get misclassified. Treat the registry as the durable configuration surface: it is what a new-bank onboarding checklist populates, and it is what the parser consults first, before any content-based inference runs.

import re
from dataclasses import dataclass, field
from enum import Enum


class SubfieldFormat(str, Enum):
    GVC = "GVC"              # Deutsche Kreditwirtschaft ?NN convention
    SLASH_TAG = "SLASH_TAG"  # SEPA-style /CODE/ structured convention


GVC_PROBE = re.compile(r"\?\d{2}")
SLASH_PROBE = re.compile(r"/[A-Z]{2,4}/")


@dataclass(frozen=True)
class SubfieldProfile:
    format: SubfieldFormat
    gvc_map: dict[str, str] = field(default_factory=dict)


def detect_profile(
    raw_86: str, bank_bic: str, registry: dict[str, SubfieldProfile]
) -> SubfieldProfile:
    if bank_bic in registry:
        return registry[bank_bic]
    if GVC_PROBE.search(raw_86):
        return SubfieldProfile(SubfieldFormat.GVC)
    if SLASH_PROBE.search(raw_86):
        return SubfieldProfile(SubfieldFormat.SLASH_TAG)
    raise ValueError("UNKNOWN_SUBFIELD_FORMAT")

Step 2 — Tokenize the raw string into ordered subfields

Both conventions repeat their delimiter arbitrarily many times, and the same code can legitimately appear more than once (?20 through ?29 are ten separate remittance-line codes, not one). The tokenizer must therefore preserve order and collect a list per code rather than overwriting on repeat. A single re.findall pass over the whole string is sufficient because both grammars are regular — no code’s value can itself contain the format’s own delimiter character, so there is no need for a stateful scanner here the way the outer MT940 tag machine needed one.

GVC_TOKEN = re.compile(r"\?(\d{2})([^?]*)")
SLASH_TOKEN = re.compile(r"/([A-Z]{2,4})/([^/]*)")


def tokenize_gvc(raw_86: str) -> dict[str, list[str]]:
    subfields: dict[str, list[str]] = {}
    for code, value in GVC_TOKEN.findall(raw_86):
        subfields.setdefault(code, []).append(value.strip())
    return subfields


def tokenize_slash(raw_86: str) -> dict[str, list[str]]:
    subfields: dict[str, list[str]] = {}
    for code, value in SLASH_TOKEN.findall(raw_86):
        subfields.setdefault(code, []).append(value.strip())
    return subfields

Step 3 — Concatenate ?20–?29 and extract counterparty identity

The DK spec caps each GVC remittance segment at 27 characters. When a segment fills that width exactly, the next segment is a hard-wrapped continuation of the same word or sentence and must be joined without an inserted space; anything shorter than 27 characters is a genuine line break and gets a space. Getting this join rule wrong is the single most common cause of a mangled counterparty descriptor reaching the matcher.

REMITTANCE_CODES = [f"{i:02d}" for i in range(20, 30)]
GVC_SEGMENT_WIDTH = 27


def concat_remittance(subfields: dict[str, list[str]]) -> str:
    parts: list[str] = []
    for code in REMITTANCE_CODES:
        for raw in subfields.get(code, []):
            if parts and len(parts[-1]) == GVC_SEGMENT_WIDTH:
                parts[-1] += raw  # hard-wrapped continuation, no inserted space
            else:
                parts.append(raw)
    return " ".join(p for p in parts if p).strip()


def extract_counterparty(
    subfields: dict[str, list[str]], fmt: SubfieldFormat
) -> tuple[str, str | None]:
    if fmt is SubfieldFormat.GVC:
        name = " ".join(subfields.get("32", []) + subfields.get("33", []))
        iban = (subfields.get("31") or [None])[0]
        return name.strip(), iban
    name = " ".join(subfields.get("BENM", []) or subfields.get("ORDP", []))
    iban = (subfields.get("IBAN") or [None])[0]
    return name.strip(), iban

?30/?31 carry the counterparty’s bank code and account (IBAN in modern feeds); ?32/?33 carry the counterparty name across up to two lines. Slash-tag feeds usually express the same identity through /BENM/ (beneficiary) or /ORDP/ (ordering party), with the IBAN either absent — because the counterparty account was already resolved upstream — or carried in a bank-specific /IBAN/ tag.

This asymmetry between the two conventions is why extract_counterparty branches on fmt rather than trying to unify the two into one lookup table: a GVC feed that omits ?31 genuinely has no IBAN for that transaction, whereas a slash-tag feed’s absent /IBAN/ usually just means the account reference lives elsewhere in the payment chain. Conflating the two would make a legitimate “no IBAN supplied” case indistinguishable from a parsing miss, which matters when a strict_mode audit later has to explain why a record has no counterparty account.

Step 4 — Resolve the end-to-end reference and normalise the descriptor

The end-to-end reference is what lets Building Deterministic Composite Keys for Hashing skip fuzzy matching entirely when it is present and trustworthy. GVC feeds carry it in ?10 (Kundenreferenz); slash-tag feeds carry it in /EREF/. Both conventions use sentinel values — NONREF, NOTPROVIDED — to signal “no reference supplied,” and those sentinels must be treated as absent, not as literal reference strings. Comparing a sentinel string byte-for-byte against another transaction’s sentinel string is a classic false-positive source: two unrelated payments both stamped NONREF are not the same payment, so an absent reference has to disqualify exact-key matching rather than accidentally enable it.

import unicodedata

_WHITESPACE = re.compile(r"\s+")
_NONREF_SENTINELS = {"NONREF", "NOTPROVIDED", ""}


def resolve_end_to_end_ref(subfields: dict[str, list[str]], fmt: SubfieldFormat) -> str:
    if fmt is SubfieldFormat.GVC:
        ref = (subfields.get("10") or [""])[0]
    else:
        ref = (subfields.get("EREF") or [""])[0]
    if ref.upper() in _NONREF_SENTINELS:
        raise ValueError("MISSING_END_TO_END_REF")
    return ref


def normalise_descriptor(text: str, max_len: int) -> str:
    if any(unicodedata.category(ch) == "Cc" and ch not in "\t\n" for ch in text):
        raise ValueError("ENCODING_CORRUPTION")
    collapsed = _WHITESPACE.sub(" ", text).strip(" \t.-")
    return collapsed[:max_len]

normalise_descriptor is deliberately the same shape of normalisation applied later, on the counterparty side of the pipeline, in Normalising Merchant Descriptors Before Matching — collapsing whitespace and stripping trailing punctuation here means the fuzzy matcher downstream is comparing already-clean strings rather than compensating for bank-specific padding.

Step 5 — Populate the canonical model and emit the audit trail

The final step wires the previous four into one function that always terminates in exactly one structured audit line, whether it succeeds or quarantines. RemittanceInfo is a pydantic model that attaches to the parent MT940Transaction.amount: Decimal record — the remittance side never touches monetary arithmetic itself, but every audit event it emits carries the same trace_id/source_hash pair as the transaction it describes, so the two records reconcile in the audit ledger by construction.

import hashlib
import logging
from datetime import datetime, timezone

from pydantic import BaseModel, field_validator

log = logging.getLogger("mt940.remittance")


class RemittanceInfo(BaseModel):
    counterparty_name: str
    counterparty_iban: str | None = None
    end_to_end_reference: str
    remittance_text: str
    source_format: SubfieldFormat

    @field_validator("counterparty_name")
    @classmethod
    def _non_empty(cls, v: str) -> str:
        return v or "UNKNOWN COUNTERPARTY"


def parse_remittance_field(
    raw_86: str,
    *,
    bank_bic: str,
    trace_id: str,
    source_hash: str,
    registry: dict[str, SubfieldProfile],
    descriptor_max_len: int = 140,
) -> RemittanceInfo:
    audit_ts = datetime.now(timezone.utc).isoformat()
    try:
        profile = detect_profile(raw_86, bank_bic, registry)
        tokenize = tokenize_gvc if profile.format is SubfieldFormat.GVC else tokenize_slash
        subfields = tokenize(raw_86)
        if not subfields:
            raise ValueError("TRUNCATED_REMITTANCE")
        remittance_text = normalise_descriptor(concat_remittance(subfields), descriptor_max_len)
        name, iban = extract_counterparty(subfields, profile.format)
        ref = resolve_end_to_end_ref(subfields, profile.format)
        info = RemittanceInfo(
            counterparty_name=name,
            counterparty_iban=iban,
            end_to_end_reference=ref,
            remittance_text=remittance_text,
            source_format=profile.format,
        )
    except ValueError as exc:
        log.info(
            "remittance.parse",
            extra={
                "trace_id": trace_id,
                "source_hash": source_hash,
                "match_decision": str(exc),
                "audit_ts": audit_ts,
            },
        )
        raise
    log.info(
        "remittance.parse",
        extra={
            "trace_id": trace_id,
            "source_hash": source_hash,
            "match_decision": "REMITTANCE_PARSED",
            "counterparty_name": info.counterparty_name,
            "has_iban": info.counterparty_iban is not None,
            "source_format": info.source_format.value,
            "audit_ts": audit_ts,
        },
    )
    return info

Note that the raised ValueError string doubles as the match_decision value on the quarantine path — the audit log and the exception taxonomy speak the same vocabulary, so a downstream dashboard can group failures by code without a separate mapping table.

Configuration boundary table

Verification and testing

A hand-built fixture with a deliberately 27-character-wide ?20 segment is the sharpest test: it proves the hard-wrap join rule from Step 3 fires correctly instead of inserting a spurious space mid-word.

def test_parses_gvc_remittance_with_hard_wrap() -> None:
    raw_86 = (
        "?00051?109923471"
        "?20INVOICE 2026-0042 SETTLEMEN"  # exactly 27 chars -> hard-wrapped
        "?21T CONFIRMED"
        "?30COBADEFFXXX"
        "?31DE89370400440532013000"
        "?32ACME GMBH"
        "?33TREASURY DEPT"
    )
    registry = {"COBADEFFXXX": SubfieldProfile(SubfieldFormat.GVC)}

    info = parse_remittance_field(
        raw_86,
        bank_bic="COBADEFFXXX",
        trace_id="t-1",
        source_hash="h-1",
        registry=registry,
    )

    assert info.counterparty_name == "ACME GMBH TREASURY DEPT"
    assert info.counterparty_iban == "DE89370400440532013000"
    assert info.end_to_end_reference == "9923471"
    assert "SETTLEMENT CONFIRMED" in info.remittance_text
    assert "INVOICE 2026-0042" in info.remittance_text

A green run confirms four independent things at once: format detection resolved to GVC from the registered BIC, the 27-character segment joined without a stray space, the counterparty name assembled from both ?32 and ?33, and the reference came from ?10 rather than falling back to a composite key. Run the same fixture through a slash-tag profile (/EREF/, /BENM/, /REMI/) before onboarding any new SEPA feed, since the two tokenizers share no code path below Step 3.

Keep at least one negative fixture per troubleshooting code in the same suite — a raw_86 with no recognisable delimiter for UNKNOWN_SUBFIELD_FORMAT, one with only a bare ?00 and nothing else for TRUNCATED_REMITTANCE, one with ?10NONREF for MISSING_END_TO_END_REF, and one with an embedded control byte for ENCODING_CORRUPTION. A regression that silently swallows one of these back into a partially-populated RemittanceInfo is worse than a loud failure, because it produces a record that looks valid to everything downstream except a human auditor.

Troubleshooting

• UNKNOWN_SUBFIELD_FORMAT — neither the ?NN nor /CODE/ probe matches, and the BIC has no registry entry. Root cause: a new bank was onboarded without a profile, or the feed genuinely sends unstructured free text. Fix: register the BIC with an explicit SubfieldFormat before the feed goes live; never let auto-detection stand in for a pinned profile in production.
• TRUNCATED_REMITTANCE — the format probe matches but tokenization yields an empty dictionary. Root cause: the bank truncated :86: at a fixed byte length that cut off before the first delimiter, or an upstream continuation-join bug dropped content before it reached this stage. Fix: verify the raw line length against the SWIFT 65-character cap and re-check the :86: continuation logic in How to Parse MT940 Files in Python.
• MISSING_END_TO_END_REF — resolve_end_to_end_ref finds only a NONREF/NOTPROVIDED sentinel or an empty value. Root cause: the paying bank did not propagate the originator’s reference. Fix: fall back to a deterministic composite key hashed from value_date, amount, and remittance_text, and route the record through amount-tolerance and fuzzy matching instead of exact-reference matching.
• ENCODING_CORRUPTION — normalise_descriptor raises on a control character surviving decoding. Root cause: a latin-1 MT940 feed was decoded as utf-8 upstream, turning accented counterparty names into mojibake. Fix: confirm the encoding contract at the parser’s boundary-validation stage before remittance parsing runs; never strip the offending bytes silently, since that can truncate a legal entity name mid-character.

Related

• How to Parse MT940 Files in Python
• Parsing OFX SGML Statements in Python
• Normalising Merchant Descriptors Before Matching
• Building Deterministic Composite Keys for Hashing

Part of OFX & MT940 Parser Design within Core Architecture & Bank Feed Ingestion.