Secure API Token Management: Credential Lifecycle for Bank-Feed Ingestion

Token management is the authentication control plane that gates the very front of the reconciliation pipeline. Before any statement can be fetched, parsed, or matched, a worker must hold a valid, scope-limited credential for the institution’s API — and it must hold it without leaking it, without two workers racing to refresh it, and without ever operating on an expired one. Within the Core Architecture & Bank Feed Ingestion layer this is where OAuth 2.0 client-credential grants, mutual-TLS (mTLS) certificates, and HMAC session tokens are issued, cached, rotated, and revoked. A defect here does not produce a wrong match later — it produces a 401, a throttled feed, or, worse, a plaintext secret in a log line that turns a routine SOX walkthrough into a finding.

This page specifies the token lifecycle as a finite state machine, gives a production-grade lease-based rotation implementation in Python, and defines the calibration parameters, failure codes, and audit records a credential subsystem must emit so that every fetch the downstream pipeline performs is traceable back to the exact authentication session that authorised it. The engineering mandate is strict: a credential is auditable state, not a convenience constant, and any token that cannot be fully validated must block the request rather than be guessed at.

Prerequisites: Pipeline State Before a Token Is Acquired

Token acquisition is the first stage a fetch worker executes, but it still depends on several upstream guarantees being in place before the state machine starts:

• A provisioned key hierarchy. A master key-encryption key (KEK) per institution scope must already live in a hardware security module (HSM) or cloud KMS. The token subsystem never holds long-lived plaintext key material; it derives a per-payload data-encryption key (DEK) at use time and discards it.
• A registered scope contract. Every credential must be bound to an explicit (institution, job, ledger_entity) scope so the principle of least privilege is enforceable. A scope that has not been registered is rejected, not defaulted.
• A consistent cache tier. A strongly consistent, low-latency store (Redis or equivalent) must be reachable for both the lease lock and the encrypted token cache. Eventually-consistent stores are disqualified here because they re-introduce the refresh race the lease exists to remove.
• A bound rate-limit budget. The institution’s concurrency and throughput ceilings must be known so rotation traffic is counted against the same budget as fetch traffic — the coupling described in Best Practices for Handling Bank API Rate Limits.

If any precondition is unmet the correct behaviour is to refuse to issue a token and emit a PRECONDITION_FAILED audit event rather than proceed with a partial credential.

Algorithm and Mechanism: The Token Lifecycle as a State Machine

A credential is modelled as a finite state machine with explicit, verifiable transitions: INITIALIZED → ACTIVE → EXPIRING → ROTATING → REVOKED. Each transition is timestamped, cryptographically attributable, and routed deterministically — there is no “best effort” edge. ACTIVE → EXPIRING fires when remaining time-to-live (TTL) crosses a configurable threshold; EXPIRING → ROTATING fires only for the single worker that wins the distributed lease; every other worker blocks or serves the still-valid cached token until the new lease propagates.

The hard correctness property is the elimination of split-brain authentication. If two workers independently refresh the same scope, the institution sees concurrent grants, the older token may be invalidated mid-flight, and in-flight fetches fail non-deterministically. The lease model solves this with a single-writer guarantee: acquire a distributed lock keyed on the scope, double-check the cache after acquisition (another worker may have refreshed while this one waited), perform the refresh against the issuer’s introspection or token endpoint, atomically commit the encrypted result, then release the lock. The refresh is the only place network I/O to the auth endpoint happens, so it is also the only place rate-limit budget is consumed.

Storage uses envelope encryption. At rest, the full token object — access token, expires_at, and scope claims — is encrypted with AES-256-GCM under a DEK that is itself derived from the scope’s KEK and a fresh 96-bit nonce; the nonce is prepended to the ciphertext and the scope is bound in as the GCM associated data, so a ciphertext minted for one scope cannot be replayed under another. In transit, all exchanges use TLS 1.3 with certificate pinning. Complexity is O(1) per acquisition on the fast path (a single cache read and decrypt) and one network round trip on the slow path; the lease bounds slow-path concurrency to exactly one refresh per scope regardless of worker-pool size.

The financial-domain caveats are unforgiving. TTL arithmetic must use a monotonic-safe expires_at absolute timestamp rather than a relative countdown, so a process restart cannot resurrect an expired token. Rotation must complete before 90% of TTL elapses, leaving a grace window for retry. And revocation must be propagable within one introspection interval, because a leaked or compromised credential that lingers is a reportable control failure.

Production-Grade Python Implementation

The implementation below is a deterministic, async-native token manager. It performs lease-based single-writer rotation, derives a per-payload DEK, encrypts the full token object with AES-256-GCM, atomically commits to a strongly consistent cache, and emits a structured audit record carrying trace_id, source_hash, and a match_decision status for every credential operation. It uses Python 3.10+ type hints, pydantic for the validated token contract, and asyncio throughout.

from __future__ import annotations

import asyncio
import hashlib
import json
import logging
import os
import time
import uuid
from typing import Literal

import httpx
import redis.asyncio as redis
from cryptography.hazmat.primitives import hashes
from cryptography.hazmat.primitives.ciphers.aead import AESGCM
from cryptography.hazmat.primitives.kdf.pbkdf2 import PBKDF2HMAC
from pydantic import BaseModel, Field

audit_log = logging.getLogger("ingestion.audit")

MatchDecision = Literal["CACHE_HIT", "ROTATED", "BLOCKED", "FAILED"]


class TokenObject(BaseModel):
    """Validated credential contract committed to the cache."""

    access_token: str = Field(min_length=1)
    expires_at: float  # absolute UNIX epoch seconds, not a relative countdown
    scope: str = Field(min_length=1)
    aud: str | None = None


def _audit(scope: str, decision: MatchDecision, source_hash: str, **extra: object) -> None:
    """Emit one immutable, hash-linkable audit record per credential operation."""
    audit_log.info(
        json.dumps(
            {
                "trace_id": str(uuid.uuid4()),
                "source_hash": source_hash,
                "scope": scope,
                "match_decision": decision,
                "emitted_at": time.time(),
                **extra,
            },
            sort_keys=True,
        )
    )


class TokenManager:
    def __init__(
        self,
        redis_client: redis.Redis,
        http_client: httpx.AsyncClient,
        keks: dict[str, bytes],  # KEK per institution scope, sourced from KMS/HSM
        lease_ttl_sec: int = 30,
    ) -> None:
        self.redis = redis_client
        self.http = http_client
        self.keks = keks
        self.lease_ttl_sec = lease_ttl_sec
        self.lock_prefix = "token:lock:"
        self.cache_prefix = "token:cache:"

    def _derive_dek(self, scope: str, nonce: bytes) -> bytes:
        kdf = PBKDF2HMAC(
            algorithm=hashes.SHA256(),
            length=32,
            salt=scope.encode(),
            iterations=100_000,
        )
        return kdf.derive(self.keks[scope] + nonce)

    def _encrypt(self, scope: str, plaintext: bytes) -> bytes:
        nonce = os.urandom(12)  # 96-bit nonce; AESGCM has no generate_nonce helper
        aesgcm = AESGCM(self._derive_dek(scope, nonce))
        ciphertext = aesgcm.encrypt(nonce, plaintext, associated_data=scope.encode())
        return nonce + ciphertext  # prepend nonce for decryption

    def _decrypt(self, scope: str, blob: bytes) -> bytes:
        nonce, ciphertext = blob[:12], blob[12:]
        aesgcm = AESGCM(self._derive_dek(scope, nonce))
        return aesgcm.decrypt(nonce, ciphertext, associated_data=scope.encode())

    def _read_cache(self, scope: str, blob: bytes | None, threshold: int) -> TokenObject | None:
        if not blob:
            return None
        token = TokenObject.model_validate_json(self._decrypt(scope, blob))
        return token if token.expires_at > time.time() + threshold else None

    async def acquire_token(self, scope: str, ttl_threshold_sec: int = 300) -> str:
        cache_key = f"{self.cache_prefix}{scope}"
        lock_key = f"{self.lock_prefix}{scope}"

        # Fast path: serve a still-valid cached credential.
        token = self._read_cache(scope, await self.redis.get(cache_key), ttl_threshold_sec)
        if token is not None:
            _audit(scope, "CACHE_HIT", source_hash=_hash(token.access_token))
            return token.access_token

        # Slow path: contend for the single-writer lease.
        if not await self.redis.set(lock_key, "1", nx=True, ex=self.lease_ttl_sec):
            _audit(scope, "BLOCKED", source_hash="")
            await asyncio.sleep(0.1)
            return await self.acquire_token(scope, ttl_threshold_sec)

        try:
            # Double-check: a peer may have rotated while we waited for the lease.
            token = self._read_cache(scope, await self.redis.get(cache_key), ttl_threshold_sec)
            if token is not None:
                _audit(scope, "CACHE_HIT", source_hash=_hash(token.access_token))
                return token.access_token

            resp = await self.http.post(
                f"https://auth.{scope}.bank/oauth/token",
                data={"grant_type": "client_credentials"},
                timeout=10.0,
            )
            resp.raise_for_status()
            body = resp.json()
            token = TokenObject(
                access_token=body["access_token"],
                expires_at=time.time() + body.get("expires_in", 3600),
                scope=scope,
                aud=body.get("aud"),
            )

            ttl_seconds = max(int(token.expires_at - time.time()), 1)
            await self.redis.set(
                cache_key,
                self._encrypt(scope, token.model_dump_json().encode()),
                ex=ttl_seconds,
            )
            _audit(scope, "ROTATED", source_hash=_hash(token.access_token), ttl=ttl_seconds)
            return token.access_token

        except Exception as exc:
            _audit(scope, "FAILED", source_hash="", error=type(exc).__name__)
            raise
        finally:
            await self.redis.delete(lock_key)


def _hash(secret: str) -> str:
    """Audit records never carry a raw token — only a non-reversible digest."""
    return hashlib.sha256(secret.encode()).hexdigest()

Two production details are worth calling out explicitly. AESGCM exposes no generate_nonce class method — produce the 96-bit nonce with os.urandom(12). And the cache stores the full validated TokenObject (so expires_at and the scope claims survive a restart), never just the bare access_token. The audit record stores only a SHA-256 digest of the token, so the log is forensically useful without ever persisting the secret itself.

Configuration Rules and Threshold Calibration

Token behaviour is governed by a small set of parameters whose defaults trade safety against API budget. Tune them per institution rather than globally — a feed with a 60-minute TTL and a tight rate ceiling needs a different profile than a 24-hour mTLS session.

Multi-Dimensional Validation

A token is never validated on TTL alone. The subsystem composes three orthogonal checks before a credential is allowed to authorise a fetch, and a failure on any one quarantines the request:

• Temporal validity — expires_at must exceed the present time plus ttl_threshold_sec. This is the cheap, fast-path gate.
• Scope and audience binding — the token’s scope and aud claims must match the (institution, job, ledger_entity) the worker is acting for. A token minted for statement retrieval must never be accepted for a payment-initiation or customer-data endpoint, which is exactly the lateral-movement boundary the GCM associated-data binding enforces cryptographically.
• Cryptographic integrity — GCM authentication must succeed on decrypt; a tampered or scope-mismatched ciphertext raises rather than returns, so a corrupted cache entry can never be served as a live credential.

This composed gate is the credential analogue of the multi-signal matching used downstream, where a candidate is only accepted when amount, date window, and string similarity all agree — the same defence-in-depth philosophy applied to authentication instead of matching.

Async and High-Throughput Execution Patterns

The manager is async-native so a worker pool can fan thousands of fetches across a handful of scopes without serialising on credential I/O. The fast path is lock-free: a cache read plus an in-memory decrypt, costing one round trip to the consistent store. Only a TTL-threshold crossing pushes a single worker onto the slow path, and the lease guarantees the rest of the pool keeps serving the validated cache rather than stampeding the auth endpoint.

For streaming feeds, rotate proactively on a predictive TTL-decay schedule so the handoff to event-driven consumers carries zero refresh latency. For windowed batch runs, pool and rotate credentials during off-peak intervals so rotation traffic never competes with the fetch burst — the streaming-versus-windowed trade-off detailed in Real-Time vs Batch Ingestion. Backpressure is handled by the rate-limit coupling: when the institution’s budget nears exhaustion, fetch admission throttles first, and because rotation is counted against the same budget, the token subsystem degrades gracefully instead of triggering a credential-exhaustion cascade.

Failure Modes Specific to This Technique

Compliance and Audit Trail Requirements

Financial credential management must satisfy SOX Section 404 internal controls, GLBA data-protection mandates, and PSD2/Open Banking requirements implemented per RFC 6749. Every transition in the token state machine emits an immutable audit record, and those records are appended to a write-once-read-many (WORM) tier with hash-linked chaining so tampering is detectable. Concretely, every credential operation must emit:

• A non-reversible token reference — the SHA-256 digest shown in the implementation, never the raw secret, so the trail is useful without being a liability.
• trace_id, source_hash, and match_decision on every record, so a downstream fetch can be tied back to the exact authentication session that authorised it and replayed deterministically from a dead-letter queue.
• Scope and audience claims restricting the token to reconciliation endpoints, evidencing least privilege for the SOX walkthrough.
• Rotation timing proving credentials rotated before rotate_at_pct of TTL, with the mTLS fallback path recorded when the OAuth endpoint degraded.

Key-lifecycle practices follow NIST SP 800-57 for the KEK/DEK hierarchy and NIST SP 800-132 for PBKDF2 work factors. Credentials must never traverse an untrusted network in plaintext or persist in process memory beyond the execution context — a guarantee the parser stage relies on when it requests decryption keys through this subsystem rather than holding them itself, as noted in OFX & MT940 Parser Design. When a fetched payload spans currencies, the FX-rate endpoint it consults must be reached under the same scope-bound session that retrieved the ledger entries, so exchange-rate snapshots stay temporally aligned for Multi-Currency Ledger Mapping, and any credential anomaly that blocks a feed is escalated through Exception Routing & Human-in-the-Loop Workflows rather than silently retried.

Related

• Best Practices for Handling Bank API Rate Limits
• OFX & MT940 Parser Design
• Real-Time vs Batch Ingestion
• Multi-Currency Ledger Mapping

Part of Core Architecture & Bank Feed Ingestion.