Designing a Slack-Integrated Approval Workflow for Unmatched Reconciliation Items

When a matching cascade exhausts its confidence thresholds, the leftover unmatched ledger entries — timing breaks, vendor-mapping drift, partial settlements, FX-conversion latency — must reach a human without breaking the audit trail or stalling the pipeline. This page implements that last mile: a Slack-native approval surface that sits downstream of Threshold-Based Routing Logic and feeds the Manual Review Queue Design it belongs to. Slack acts strictly as an interaction broker; every state transition, idempotency check, and ledger mutation stays in your backend reconciliation service. The result is a workflow where a reviewer taps Approve or Reject in a channel, and the posting decision is captured as an immutable, replay-safe audit event.

Slack approval round-trip: verify, lock, mutate, audit, confirm Lifelines left to right: Reviewer in Slack, the backend Approval Gateway at /webhooks/slack/approval, the Redis idempotency store, the ledger, and the append-only audit sink. The reviewer's Approve tap sends a signed POST to the gateway. The gateway verifies the Slack HMAC over the raw bytes with a 300-second skew guard, then runs SET NX on a deterministic action token; an existing key short-circuits to already_processed. On a fresh lock the gateway posts the ledger mutation, receives a LEDGER reference, appends an immutable AUDIT_COMMIT before responding, and returns HTTP 200 with an ephemeral confirmation to Slack. Reviewer Slack channel Approval Gateway /webhooks/slack/approval Idempotency Redis · SET NX Ledger posting decision Audit Sink append-only · WORM taps Approve → signed POST verify HMAC over raw bytes 300s replay window · constant-time compare SET NX action token item + reviewer + action + window acquired → proceed  ·  exists → already_processed post mutation (approve only) returns ledger_mutation_ref append AUDIT_COMMIT  ·  before HTTP 200 trace_id · source_hash · match_decision — immutable HTTP 200 · ephemeral confirmation Slack brokers the interaction; every state transition, lock, and ledger write stays in the backend.

Prerequisites

Confirm the upstream pipeline state and platform configuration before wiring the approval surface:

Step-by-Step Implementation

Step 1 — Model the unmatched item and the audit record

Define strict Pydantic v2 schemas so malformed exceptions never enter the approval path. Monetary values use Decimal, and the audit record carries the trace_id, source_hash, and match_decision that every code path in this pipeline must emit.

python
import hashlib
import logging
from decimal import Decimal
from typing import Optional
from pydantic import BaseModel, Field

audit_logger = logging.getLogger("finops.reconciliation.slack_approval")

class UnmatchedItem(BaseModel):
    item_id: str
    amount: Decimal
    aging_days: int
    confidence_score: float = Field(ge=0.0, le=1.0)
    counterparty_risk_tier: str = Field(pattern="^(low|medium|high|critical)$")

class AuditRecord(BaseModel):
    event_id: str
    trace_id: str
    item_id: str
    action: str                       # approved | rejected | escalated
    reviewer_id: str
    source_hash: str                  # hash of the canonical item payload
    match_decision: str               # routing tier that produced this review
    timestamp: float
    idempotency_key: str
    ledger_mutation_ref: Optional[str] = None

def source_hash(item: UnmatchedItem) -> str:
    canonical = f"{item.item_id}|{item.amount}|{item.counterparty_risk_tier}"
    return hashlib.sha256(canonical.encode("utf-8")).hexdigest()

def persist_audit(record: AuditRecord) -> None:
    """Append-only sink. Never UPDATE; corrections are new linked events."""
    audit_logger.info(
        "AUDIT_COMMIT trace_id=%s source_hash=%s match_decision=%s payload=%s",
        record.trace_id, record.source_hash, record.match_decision,
        record.model_dump(mode="json"),
    )

Step 2 — Verify the Slack request signature over raw bytes

Slack’s HMAC check runs on the raw POST body, not parsed JSON or a header. Read await request.body() before any request.json() call — FastAPI caches the body so the later parse still works. The 300-second window guards against replay.

python
import hmac
import time
from fastapi import Request, HTTPException

async def verify_slack_signature(request: Request, signing_secret: str) -> bytes:
    ts = request.headers.get("X-Slack-Request-Timestamp", "")
    slack_sig = request.headers.get("X-Slack-Signature", "")
    if not ts or not slack_sig:
        raise HTTPException(status_code=401, detail="Missing Slack signature headers")
    if abs(time.time() - float(ts)) > 300:
        raise HTTPException(status_code=401, detail="Stale timestamp (replay guard)")

    raw_body = await request.body()            # bytes — read before any JSON parse
    basestring = f"v0:{ts}:{raw_body.decode('utf-8')}"
    expected = "v0=" + hmac.new(
        signing_secret.encode("utf-8"),
        basestring.encode("utf-8"),
        hashlib.sha256,
    ).hexdigest()
    if not hmac.compare_digest(expected, slack_sig):
        raise HTTPException(status_code=401, detail="Invalid Slack signature")
    return raw_body

Signature verification follows Slack’s request-verification protocol; hmac.compare_digest keeps the comparison constant-time.

Step 3 — Route deterministically and build the approval card

Routing is the same deterministic gate enforced by Threshold-Based Routing Logic: material, aged, low-confidence, or high-risk items get an immediate Slack card; everything else folds into a digest handled by Batch Approval Automation. The match_decision string returned here is stamped onto every audit record.

python
ROUTING_CONFIG = {
    "materiality_threshold": Decimal("10000.00"),
    "aging_sla_days": 5,
    "min_confidence": 0.85,
}

def evaluate_routing(item: UnmatchedItem) -> str:
    if item.amount >= ROUTING_CONFIG["materiality_threshold"]:
        return "immediate"
    if item.aging_days >= ROUTING_CONFIG["aging_sla_days"]:
        return "immediate"
    if item.confidence_score < ROUTING_CONFIG["min_confidence"]:
        return "immediate"
    if item.counterparty_risk_tier in ("critical", "high"):
        return "immediate"
    return "batch"

def build_approval_blocks(item: UnmatchedItem) -> list[dict]:
    return [
        {"type": "section", "text": {
            "type": "mrkdwn",
            "text": (f"*Unmatched item* `{item.item_id}`\n"
                     f"Amount: ${item.amount} · Aging: {item.aging_days}d · "
                     f"Confidence: {item.confidence_score:.2f} · "
                     f"Tier: {item.counterparty_risk_tier}")}},
        {"type": "actions", "elements": [
            {"type": "button", "action_id": item.item_id,
             "text": {"type": "plain_text", "text": "Approve"},
             "style": "primary", "value": "approve"},
            {"type": "button", "action_id": item.item_id,
             "text": {"type": "plain_text", "text": "Reject"},
             "style": "danger", "value": "reject"},
        ]},
    ]

Step 4 — Handle the block action with idempotency and audit

The reviewer’s tap returns a block-action payload. Generate a deterministic action token scoped to item_id + reviewer_id + action + time_window, gate it through your idempotency store, then write the audit record before returning HTTP 200 so the attempted action is always traceable.

python
import secrets
import uuid
from fastapi import FastAPI

app = FastAPI(title="FinOps Reconciliation Approval Gateway")
SLACK_SIGNING_SECRET = "load-from-os.environ-in-production"

def idempotency_key(item_id: str, reviewer_id: str, action: str) -> str:
    raw = f"{item_id}:{reviewer_id}:{action}:{int(time.time() // 300)}"
    return hashlib.sha256(raw.encode()).hexdigest()

@app.post("/webhooks/slack/approval")
async def handle_slack_approval(request: Request):
    trace_id = str(uuid.uuid4())
    await verify_slack_signature(request, SLACK_SIGNING_SECRET)   # raw-bytes HMAC first

    body = await request.json()
    try:
        action = body["actions"][0]
        action_value = action["value"]                 # approve | reject
        item_id = action.get("action_id", "unknown")
        reviewer_id = body["user"]["id"]
    except (KeyError, IndexError) as exc:
        raise HTTPException(status_code=400, detail=f"Malformed Slack payload: {exc}")

    key = idempotency_key(item_id, reviewer_id, action_value)
    # if not await redis.set(key, "1", nx=True, ex=86400):
    #     return {"status": "already_processed", "idempotency_key": key}

    if action_value not in ("approve", "reject"):
        raise HTTPException(status_code=400, detail=f"Unsupported action: {action_value!r}")

    ledger_ref = f"LEDGER-{secrets.token_hex(8).upper()}" if action_value == "approve" else None
    record = AuditRecord(
        event_id=secrets.token_hex(16),
        trace_id=trace_id,
        item_id=item_id,
        action="approved" if action_value == "approve" else "rejected",
        reviewer_id=reviewer_id,
        source_hash=hashlib.sha256(item_id.encode()).hexdigest(),
        match_decision="immediate",
        timestamp=time.time(),
        idempotency_key=key,
        ledger_mutation_ref=ledger_ref,
    )
    persist_audit(record)            # audit BEFORE the response
    return {"status": "processed", "action": action_value, "trace_id": trace_id}

Step 5 — Wire the escalation fallback chain

A stalled reviewer must never silently expire an item. Layer a time-based escalation on top of the queue, and degrade gracefully when Slack itself fails — the same resilience contract enforced by Fallback Chain Configuration.

  1. T+24h reminder — re-ping the assigned reviewer in-thread.
  2. T+48h escalation — reassign to a secondary approver or manager channel.
  3. T+72h dead-letter — route to a compliance queue, halt auto-posting, raise a high-severity alert.
  4. API fallback — on rate_limited or channel_not_found, enqueue the payload in a Redis-backed dead-letter queue with exponential backoff and emit a metric; default the item to manual_review with fallback_mode=true so no approval is ever lost.

Configuration Boundary Table

Parameter Default Valid range Notes
materiality_threshold 10000.00 0 – 1e9 Decimal; above this an item always gets an immediate card.
aging_sla_days 5 1 – 90 Days before an unmatched item forces immediate review.
min_confidence 0.85 0.50 – 0.99 Match scores below this never auto-batch.
idempotency_window_s 300 60 – 900 Time bucket for the action token; also the replay guard window.
idempotency_ttl_s 86400 3600 – 604800 How long the SET NX key blocks duplicate taps.
signature_max_skew_s 300 60 – 300 Slack-recommended ceiling; do not raise above 300.
reminder_after_h / escalate_after_h / dead_letter_after_h 24 / 48 / 72 1 – 168 Escalation rungs; must be strictly increasing.
backoff_max_s 30 5 – 300 Cap on min(2**n + jitter, cap) retry backoff.

Verification and Testing

Validate the workflow end-to-end against a deterministic fixture rather than a live channel:

  • Signature unit test. Build a known body, compute v0=<hmac> with a test secret, and assert verify_slack_signature accepts a fresh timestamp and rejects one older than 300s. Tamper one byte of the body and assert a 401.
  • Routing fixture. Feed a small ledger fixture — one $25,000 item, one 12-day-aged item, one 0.40-confidence item, and one clean $10 item — and assert the first three return immediate and the last returns batch.
  • Idempotency replay. POST the same block-action payload twice within the window; with the redis.set(..., nx=True) guard uncommented, assert the second call returns already_processed and that exactly one AUDIT_COMMIT is emitted.
  • Audit completeness. Assert every emitted AuditRecord carries a non-empty trace_id, source_hash, and match_decision, and that approved actions have a ledger_mutation_ref while rejected actions do not.
  • Fallback simulation. Stub the Slack client to raise rate_limited; assert the item lands in the dead-letter queue with fallback_mode=true and a backoff metric is emitted.

Troubleshooting

  • INVALID_SIGNATURE (401 on every callback). The body was parsed before hashing, or a proxy re-encoded it. Read await request.body() first and hash the exact bytes; never reconstruct the body from parsed JSON.
  • STALE_TIMESTAMP (intermittent 401s). Container clock drift beyond 300s. Run NTP/chrony on the workers; keep signature_max_skew_s at 300 rather than widening it.
  • DUPLICATE_APPROVAL (two ledger refs for one item). Idempotency disabled or scoped wrong. Restore the atomic SET NX and key on item_id + reviewer_id + action + time_window, not on the Slack message_ts.
  • CHANNEL_NOT_FOUND (cards never appear). The bot was removed from the target channel or the ID rotated. Catch the error, route to the dead-letter queue, and alert — do not drop the item.
  • MALFORMED_PAYLOAD (400 on callback). A non-button interaction (modal submit, overflow menu) hit the same endpoint. Branch on body["type"] and actions[0]["type"] before indexing into actions.

Part of Manual Review Queue Design · up to Exception Routing & Human-in-the-Loop Workflows.