Best Practices for Handling Bank API Rate Limits in Automated Reconciliation

Banking APIs enforce strict rate limits to protect their infrastructure and fairly allocate throughput across institutional and retail clients. In an automated reconciliation pipeline, exceeding those thresholds is not a transient network blip — it is a deterministic data-integrity failure that produces partial statement syncs, stale FX states, and reconciliation drift that compounds across accounting periods. This scenario sits at the fetch boundary of Core Architecture & Bank Feed Ingestion: every request a worker makes is metered against the same budget that Secure API Token Management spends on credential rotation, so rate-limit handling must be designed as a first-class constraint on the ingestion scheduler rather than a downstream exception handler. The sections below give a step-by-step, audit-ready implementation that keeps network constraints from ever corrupting ledger state.

Prerequisites

Before this fetch loop runs, the upstream pipeline state must already be in place. Confirm each dependency:

• A valid, scope-limited credential is held and proactively refreshed by Secure API Token Management, so authentication never re-authenticates mid-batch.
• The institution’s published ceilings — X-RateLimit-Limit, the window length, and the Retry-After semantics — are recorded per (institution, endpoint) scope.
• A strongly consistent store (PostgreSQL plus Redis or equivalent) is reachable for cursor checkpoints and the shared token-bucket state.
• The ingestion cadence — streaming webhook versus windowed poll — is decided per feed, as covered in Real-Time vs Batch Ingestion.
• A versioned FX-rate snapshot exists so multi-currency mapping never triggers a second rate-limited dependency at fetch time.
• A dead-letter queue and alert route into Exception Routing & Human-in-the-Loop Workflows for credentials or feeds that exhaust their retry budget.

If any precondition is unmet, refuse to start the fetch and emit a PRECONDITION_FAILED audit event rather than burn quota on a request that cannot be safely committed.

Step-by-Step Implementation

Step 1 — Model the rate budget from response headers

Decouple network throughput from ledger-state progression by modelling the limit as a hard boundary on the scheduler. Implement a sliding-window token bucket whose capacity tracks the bank’s X-RateLimit-Remaining header rather than a hardcoded guess, so burst ingestion cannot exhaust the quota during high-volume settlement windows like month-end payroll runs. An asyncio.Semaphore bounds in-flight concurrency; the header reading corrects the local count after every response.

import asyncio
import time
import logging

logger = logging.getLogger("finops.rate_limiter")


class HeaderDrivenBucket:
    def __init__(self, limit: int, window_seconds: float) -> None:
        self._limit = limit
        self._window = window_seconds
        self._semaphore = asyncio.Semaphore(limit)
        self._remaining = limit
        self._window_start = time.monotonic()

    async def acquire(self) -> None:
        await self._semaphore.acquire()
        now = time.monotonic()
        if now - self._window_start >= self._window:
            self._remaining = self._limit
            self._window_start = now

    def release(self, headers: dict[str, str], *, trace_id: str) -> None:
        # Reconcile the local view with the bank's authoritative count.
        if "X-RateLimit-Remaining" in headers:
            self._remaining = int(headers["X-RateLimit-Remaining"])
        logger.info(
            "bucket.release",
            extra={"trace_id": trace_id, "source_hash": None,
                   "match_decision": "rate_budget_ok", "remaining": self._remaining},
        )
        self._semaphore.release()

Step 2 — Detect silent throttling, not just 429s

Rate-limit exhaustion usually surfaces as an HTTP 429, but bank APIs frequently throttle silently: a 200 OK carrying a truncated payload, or a connection reset during a long poll. Validate response completeness against the declared record count or a checksum before committing anything to the staging ledger. When truncation is detected, halt, log the sequence gap, and schedule a targeted backfill — never retry blindly.

import hashlib


class PayloadTruncated(Exception):
    pass


def verify_payload(body: bytes, declared_count: int, parsed_count: int,
                   *, trace_id: str) -> str:
    source_hash = hashlib.sha256(body).hexdigest()
    if parsed_count != declared_count:
        logger.warning(
            "payload.truncated",
            extra={"trace_id": trace_id, "source_hash": source_hash,
                   "match_decision": "halt_backfill",
                   "expected": declared_count, "got": parsed_count},
        )
        raise PayloadTruncated(f"expected {declared_count}, parsed {parsed_count}")
    logger.info(
        "payload.complete",
        extra={"trace_id": trace_id, "source_hash": source_hash,
               "match_decision": "commit_ok"},
    )
    return source_hash

Step 3 — Back off with jitter behind a circuit breaker

Avoid naive time.sleep() loops and unbounded retry decorators. Honour Retry-After when the bank supplies it; otherwise apply capped exponential backoff with jitter so a fleet of workers does not retry in lockstep. After N consecutive throttles or truncations, open a circuit breaker, hand the feed to the dead-letter queue, and alert — quota damage stops being self-inflicted.

import random
import httpx


class CircuitOpen(Exception):
    pass


async def fetch_page(client: httpx.AsyncClient, bucket: HeaderDrivenBucket,
                     url: str, cursor: str, *, trace_id: str,
                     max_retries: int = 3) -> httpx.Response:
    consecutive_429 = 0
    for attempt in range(max_retries):
        await bucket.acquire()
        try:
            resp = await client.get(url, params={"cursor": cursor}, timeout=15.0)
            if resp.status_code == 429:
                consecutive_429 += 1
                retry_after = float(resp.headers.get("Retry-After", 2 ** attempt))
                backoff = retry_after + random.uniform(0, 1.0)
                logger.warning(
                    "rate_limited",
                    extra={"trace_id": trace_id, "source_hash": None,
                           "match_decision": "backoff", "sleep_s": round(backoff, 2)},
                )
                if consecutive_429 >= max_retries:
                    raise CircuitOpen(f"{consecutive_429} consecutive 429s on {url}")
                await asyncio.sleep(backoff)
                continue
            resp.raise_for_status()
            return resp
        finally:
            bucket.release(dict(resp.headers) if "resp" in dir() else {}, trace_id=trace_id)
    raise RuntimeError(f"exhausted {max_retries} retries for {url}")

Step 4 — Checkpoint the cursor idempotently

When a token expires or a limit is hit mid-ingestion, the pipeline must resume from the last acknowledged position without re-posting transactions. Persist the cursor inside the same database transaction that stages the page, and make every ledger write an idempotent upsert (INSERT ... ON CONFLICT DO UPDATE) keyed on the bank’s transaction id. A rate-limit retry must never generate a duplicate posting.

import asyncpg


async def stage_page(conn: asyncpg.Connection, rows: list[dict], next_cursor: str,
                     feed_id: str, *, trace_id: str, source_hash: str) -> None:
    async with conn.transaction():
        await conn.executemany(
            """
            INSERT INTO staging_ledger (txn_id, value_date, amount, currency, source_hash)
            VALUES ($1, $2, $3, $4, $5)
            ON CONFLICT (txn_id) DO UPDATE SET source_hash = EXCLUDED.source_hash
            """,
            [(r["txn_id"], r["value_date"], r["amount"], r["currency"], source_hash)
             for r in rows],
        )
        # Commit the cursor only after the page is durably staged.
        await conn.execute(
            "UPDATE feed_state SET cursor = $1 WHERE feed_id = $2", next_cursor, feed_id,
        )
    logger.info(
        "cursor.checkpoint",
        extra={"trace_id": trace_id, "source_hash": source_hash,
               "match_decision": "checkpoint_ok", "rows": len(rows)},
    )

Step 5 — Read FX from a cached snapshot, never a live lookup

Multi-currency mapping introduces a second rate-limited dependency: the FX-rate endpoint. Never couple transaction ingestion to a live rate fetch. Maintain a local, versioned snapshot refreshed on a fixed schedule, and apply the rate for each transaction’s value date so ledger balances stay deterministic. This isolation keeps FX throttling from stalling the feed and feeds clean values into Multi-Currency Ledger Mapping.

from decimal import Decimal
from datetime import date


def to_base_currency(amount: Decimal, currency: str, value_date: date,
                     fx_snapshot: dict[tuple[str, date], Decimal],
                     base: str = "USD", *, trace_id: str) -> Decimal:
    if currency == base:
        return amount
    rate = fx_snapshot.get((currency, value_date))
    if rate is None:
        logger.warning(
            "fx.missing_rate",
            extra={"trace_id": trace_id, "source_hash": None,
                   "match_decision": "quarantine", "ccy": currency, "vd": str(value_date)},
        )
        raise KeyError(f"no cached FX for {currency} @ {value_date}")
    return (amount * rate).quantize(Decimal("0.01"))

Configuration Boundaries

Verification and Testing

Validate the loop against a deterministic fixture before pointing it at a live feed:

• Bucket math. Drive 1,000 simulated requests through HeaderDrivenBucket with a mocked X-RateLimit-Remaining and assert in-flight concurrency never exceeds max_requests_per_window.
• Truncation guard. Feed verify_payload a body whose parsed_count is one short of declared_count and assert it raises PayloadTruncated and emits a halt_backfill audit record.
• Idempotency. Replay the same staged page twice through stage_page against a fixture database and assert the row count and source_hash are unchanged — no duplicate postings.
• Backoff honours headers. Mock a 429 with Retry-After: 5 and assert the slept interval is 5 s plus jitter, then assert the breaker opens on the third consecutive 429.
• FX determinism. Run a multi-currency fixture through to_base_currency twice and assert byte-identical Decimal output, confirming the cached snapshot is the only rate source.

A passing run leaves the staging ledger matching the fixture’s expected transaction count, with one cursor checkpoint per committed page and zero plaintext credentials in the logs.

Troubleshooting

• RATE_BUDGET_EXHAUSTED (HTTP 429 storm). The local bucket drifted above the bank’s count, usually from multiple workers sharing one scope without a shared store. Move the bucket state into Redis and reconcile from X-RateLimit-Remaining on every response.
• SILENT_TRUNCATION (200 OK, short payload). The bank throttled by trimming records. verify_payload should already halt the commit; if drift still reaches the ledger, the declared count is being read from the wrong envelope field — pin it to the statement trailer, not the page header.
• DUPLICATE_POSTING (constraint or balance mismatch). A retry re-staged a page whose cursor had not committed. Confirm the upsert key is the bank txn_id and that the cursor UPDATE shares the same transaction as the insert.
• FX_RATE_MISSING (quarantine on map). The value date has no cached rate, typically a weekend or holiday gap. Carry forward the last published rate within the snapshot and widen fx_snapshot_ttl_seconds for low-liquidity pairs.
• CIRCUIT_OPEN (feed handed to DLQ). Sustained throttling tripped the breaker. Inspect whether rotation traffic from token refresh is competing for the same budget; if so, route refreshes to a dedicated low-frequency queue and escalate the feed through Exception Routing & Human-in-the-Loop Workflows.

Related

• Real-Time vs Batch Ingestion
• Multi-Currency Ledger Mapping
• OFX & MT940 Parser Design

Part of Secure API Token Management, within Core Architecture & Bank Feed Ingestion.