Fallback Chain Configuration for Automated Reconciliation Exceptions

A fallback chain is the deterministic execution backbone that decides what happens to a ledger entry after the matching cascade has failed to produce a confident result. Within the Exception Routing & Human-in-the-Loop Workflows stage of the pipeline, it sits between the probabilistic matchers and the people: when an exact-hash match misses, a fuzzy score lands below tolerance, or a counterparty reference is absent, the chain cascades the exception through an ordered series of progressively more expensive resolution strategies — tolerance write-offs, master-data enrichment, historical pattern lookup — and escalates to a human queue only when every automated node is exhausted. The objective is to guarantee that no transaction is ever silently dropped or stuck in an ambiguous state, while every transition it makes is replayable for SOX, GAAP, and IFRS audit purposes. This page specifies the algorithm, a production Python implementation, the configuration surface, and the failure modes that distinguish a robust chain from a fragile retry loop.

Where this runs in the reconciliation pipeline

The matching cascade upstream is a confidence funnel. A normalised payload is first tested by exact Exact Match & Hash Comparison; survivors drop into tolerance-and-similarity scoring governed by Date-Window & Amount Tolerance Rules; and the residue — entries the matchers could not resolve with sufficient confidence — is precisely what the fallback chain consumes. In other words, the chain’s input is the unmatched and partially-matched tail of the Transaction Matching Algorithms & Logic cascade, annotated with the reason each matcher declined.

This positioning matters because the chain must never re-run the matchers’ happy path. Its job is recovery, not matching. Each node encodes a recovery hypothesis — “perhaps this is a sub-cent rounding artefact”, “perhaps the amounts differ only by a stale FX rate”, “perhaps the counterparty string is a known alias” — and either confirms the hypothesis and resolves, or declines and hands the exception to the next node unchanged.

Prerequisites: the upstream state the chain assumes

The fallback chain is only deterministic if its input is. Before an exception enters the first node, the following must already be true:

• Normalised schema. Every exception carries a canonical record: Decimal amounts (never float), an ISO 4217 currency code, an ISO 8601 UTC timestamp, a source_hash over the immutable fields, and the decline_reason emitted by the matcher that rejected it.
• A frozen match attempt. The chain receives the best partial match the cascade found (or None), including each candidate’s score, so a node can reason about why confidence was insufficient rather than re-deriving it.
• An idempotency key. Derived from source_hash plus the chain’s configuration version, so a replay after a crash cannot double-post or double-escalate.
• A trace context. A trace_id that threads through every node and every audit record, joining the exception to its originating reconciliation run.

If any of these are missing the exception should fail fast with MISSING_PRECONDITION rather than entering the chain, because a chain fed unnormalised input produces non-reproducible decisions — the cardinal sin in an audited system.

Algorithm and mechanism

Formally, the chain is a finite-state machine over an ordered list of resolution nodes N = [n₀, n₁, … n_k]. An exception holds a state in {NEW, EVALUATING, RESOLVED, ESCALATED, FAILED}. Starting at n₀, each node returns one of three verdicts:

• RESOLVE — the node is confident; it attaches a resolution payload and the FSM transitions to RESOLVED. The cascade stops.
• FALLTHROUGH — the node’s hypothesis does not apply; the FSM advances to n_{i+1} with the exception unmodified except for an appended audit event.
• RETRY_LATER — a transient dependency (FX API, master-data service) is unavailable; the node trips its circuit breaker and the exception is parked for a bounded backoff rather than failing through to escalation.

When the list is exhausted with no RESOLVE, the FSM transitions to ESCALATED and hands off to the human queue. Complexity is O(k) node evaluations per exception in the worst case, with k typically 3–6; the dominant cost is not the traversal but the I/O inside enrichment nodes, which is why ordering matters — cheap, high-probability nodes go first so the common case exits early.

Two financial-domain caveats shape the design. First, arithmetic must be exact: tolerance and write-off nodes compare Decimal deltas with explicit quantisation, because the Python decimal module is the only way to make rounding rules match an institution’s accounting policy. Second, node ordering is a control, not a tuning knob: changing it changes which resolution an exception receives, so the order is part of the version-controlled configuration and every change is peer-reviewed.

Production-grade Python implementation

The implementation below models the chain with pydantic for contract enforcement, asyncio for the I/O-bound enrichment nodes, and a structured audit logger that emits trace_id, source_hash, and match_decision on every transition. Nodes are pure: given an exception they return a verdict and never mutate global state.

from __future__ import annotations

import asyncio
import logging
from decimal import Decimal
from enum import Enum
from typing import Awaitable, Callable, Protocol

from pydantic import BaseModel, ConfigDict, Field

audit_log = logging.getLogger("reconciliation.fallback")


class Verdict(str, Enum):
    RESOLVE = "RESOLVE"
    FALLTHROUGH = "FALLTHROUGH"
    RETRY_LATER = "RETRY_LATER"


class State(str, Enum):
    NEW = "NEW"
    EVALUATING = "EVALUATING"
    RESOLVED = "RESOLVED"
    ESCALATED = "ESCALATED"
    FAILED = "FAILED"


class Exception_(BaseModel):
    """A normalised, frozen reconciliation exception entering the chain."""
    model_config = ConfigDict(frozen=True)

    trace_id: str
    source_hash: str
    amount: Decimal
    currency: str = Field(min_length=3, max_length=3)
    counterparty: str | None
    decline_reason: str
    best_score: Decimal = Decimal("0")


class NodeResult(BaseModel):
    verdict: Verdict
    node: str
    detail: str = ""
    resolution: dict[str, str] | None = None


class FallbackNode(Protocol):
    name: str
    async def evaluate(self, exc: Exception_) -> NodeResult: ...


def _emit(exc: Exception_, node: str, decision: str, detail: str = "") -> None:
    audit_log.info(
        "fallback_transition",
        extra={
            "trace_id": exc.trace_id,
            "source_hash": exc.source_hash,
            "match_decision": decision,
            "node": node,
            "detail": detail,
        },
    )


class FallbackChain:
    def __init__(self, nodes: list[FallbackNode]) -> None:
        if not nodes:
            raise ValueError("a fallback chain requires at least one node")
        self._nodes = nodes

    async def run(self, exc: Exception_) -> tuple[State, NodeResult | None]:
        _emit(exc, node="<chain>", decision="EVALUATING",
              detail=f"reason={exc.decline_reason}")
        for node in self._nodes:
            result = await node.evaluate(exc)
            _emit(exc, node=node.name,
                  decision=result.verdict.value, detail=result.detail)

            if result.verdict is Verdict.RESOLVE:
                return State.RESOLVED, result
            if result.verdict is Verdict.RETRY_LATER:
                # Park for bounded backoff; the scheduler re-enqueues.
                return State.EVALUATING, result
            # FALLTHROUGH: advance with the exception unmodified.

        _emit(exc, node="<chain>", decision="ESCALATED",
              detail="all nodes exhausted")
        return State.ESCALATED, None

A representative leaf node — the sub-cent tolerance write-off — shows the contract a node must honour: exact arithmetic, a single verdict, and no side effects beyond its returned resolution.

class ToleranceWriteOffNode:
    name = "tolerance_writeoff"

    def __init__(self, band: Decimal) -> None:
        self._band = band  # e.g. Decimal("0.05")

    async def evaluate(self, exc: Exception_) -> NodeResult:
        residual = abs(exc.amount).quantize(Decimal("0.01"))
        if residual <= self._band:
            return NodeResult(
                verdict=Verdict.RESOLVE,
                node=self.name,
                detail=f"residual {residual} within band {self._band}",
                resolution={"action": "write_off", "amount": str(residual)},
            )
        return NodeResult(
            verdict=Verdict.FALLTHROUGH,
            node=self.name,
            detail=f"residual {residual} exceeds band {self._band}",
        )

An enrichment node that calls an external master-data service demonstrates the RETRY_LATER path and the circuit breaker that keeps a slow dependency from cascading every exception straight to escalation.

class MasterDataEnrichNode:
    name = "master_data_enrich"

    def __init__(self, lookup: Callable[[str], Awaitable[str | None]],
                 breaker_open: Callable[[], bool]) -> None:
        self._lookup = lookup
        self._breaker_open = breaker_open

    async def evaluate(self, exc: Exception_) -> NodeResult:
        if self._breaker_open():
            return NodeResult(verdict=Verdict.RETRY_LATER, node=self.name,
                              detail="master-data circuit breaker open")
        if exc.counterparty is None:
            return NodeResult(verdict=Verdict.FALLTHROUGH, node=self.name,
                              detail="no counterparty to enrich")
        try:
            canonical = await asyncio.wait_for(
                self._lookup(exc.counterparty), timeout=2.0)
        except asyncio.TimeoutError:
            return NodeResult(verdict=Verdict.RETRY_LATER, node=self.name,
                              detail="lookup timeout")
        if canonical:
            return NodeResult(
                verdict=Verdict.RESOLVE, node=self.name,
                detail=f"resolved alias -> {canonical}",
                resolution={"action": "rematch", "counterparty": canonical},
            )
        return NodeResult(verdict=Verdict.FALLTHROUGH, node=self.name,
                          detail="no canonical alias found")

Configuration rules and threshold calibration

The chain’s behaviour is entirely controlled by its node list and each node’s parameters. Treat this table as a starting point and tune against a labelled backtest of historical exceptions, not intuition.

The most consequential calibration is node order. Place deterministic, cheap, high-hit-rate nodes (tolerance, FX normalisation) before I/O-bound enrichment nodes so the majority of exceptions resolve before any network call. Because reordering changes outcomes, the order is calibrated alongside the routing bands defined in Threshold-Based Routing Logic, not independently of them.

Multi-dimensional validation

A single node verdict is rarely sufficient on its own; robust chains combine constraints so that a RESOLVE is defensible under audit. A tolerance write-off, for example, should fire only when the amount delta is within band and the date delta is within the window and the counterparty similarity clears a floor — the same multi-axis logic the matchers use, re-applied as a guard. This keeps the chain from “rescuing” an exception that is genuinely two different transactions that happen to differ by a few cents.

In practice each node composes the orthogonal signals it needs: the amount residual from exact Decimal arithmetic, the temporal delta validated against Date-Window & Amount Tolerance Rules, and a string similarity score (rapidfuzz.fuzz.token_set_ratio) over the enriched counterparty name. A node resolves only when all dimensions agree; any single dimension out of bounds forces FALLTHROUGH. This conjunctive design is what makes an automated resolution survive a regulator asking “why did the machine close this exception?”.

Async and high-throughput execution patterns

Reconciliation runs produce exceptions in bursts at month-end close, so the chain must process them concurrently without losing ordering guarantees within a single account. The pattern is a bounded worker pool draining an asyncio.Queue, with concurrency capped to protect downstream enrichment services and a semaphore enforcing backpressure.

async def drain(chain: FallbackChain, queue: asyncio.Queue[Exception_],
                concurrency: int = 16) -> None:
    sem = asyncio.Semaphore(concurrency)

    async def worker(exc: Exception_) -> None:
        async with sem:
            state, result = await chain.run(exc)
            if state is State.ESCALATED:
                await enqueue_for_review(exc)          # human queue
            elif state is State.RESOLVED and result:
                await post_resolution(exc, result)     # ledger post
            audit_log.info(
                "fallback_complete",
                extra={"trace_id": exc.trace_id,
                       "source_hash": exc.source_hash,
                       "match_decision": state.value},
            )

    tasks: list[asyncio.Task[None]] = []
    while not queue.empty():
        exc = await queue.get()
        tasks.append(asyncio.create_task(worker(exc)))
    await asyncio.gather(*tasks)

Key throughput rules: partition the input queue by account or counterparty so that within a partition exceptions are processed in order (preventing two adjustments to the same ledger line from racing); size the semaphore to the slowest enrichment dependency’s safe concurrency, not the CPU count; and let RETRY_LATER results flow to a delay queue with exponential backoff rather than blocking a worker. When downstream approval capacity saturates, backpressure must propagate upstream into the matchers’ output rather than letting the queue grow unbounded — the same discipline applied in Batch Approval Automation.

Failure modes specific to the fallback chain

The unifying principle is fail towards a human, never towards a silent resolution. Any condition the chain cannot deterministically handle must surface in the review queue with full context, which is the contract defined by Manual Review Queue Design.

Compliance and audit trail requirements

Every transition the chain makes is evidence. For SOX traceability, each node evaluation must emit an immutable audit record carrying, at minimum: trace_id, source_hash, the config_version in force, the node name, the match_decision (verdict), and a UTC timestamp. Resolutions additionally record the resolution action and the operator or service principal that the action is attributable to — for automated nodes that is the signed configuration version, which makes the machine’s decision as reconstructable as a human’s.

Three properties make the trail audit-ready. Immutability: records are append-only and content-addressed, so tampering is detectable. Reproducibility: because input is frozen and node order is versioned, re-running an archived exception against its stamped config_version reproduces the identical decision. Separation of duties: the matchers, the chain, and the ledger-posting step are distinct principals, so no single component both decides and posts without an independent record. When all automated nodes are exhausted, the escalation payload handed to the human queue inherits this complete lineage, and the downstream SLA clock — instrumented in Tracking dispute resolution SLAs in Python — starts from the chain’s final ESCALATED event, giving auditors an unbroken timeline from ingestion to resolution.

Related

• Threshold-Based Routing Logic
• Manual Review Queue Design
• Batch Approval Automation
• Tracking dispute resolution SLAs in Python
• Transaction Matching Algorithms & Logic

Part of Exception Routing & Human-in-the-Loop Workflows.