SOX Evidence Packaging for Reconciliation Controls Testing

An auditor testing a Sarbanes-Oxley Section 404 control does not want your dashboard, your Slack thread, or a screenshot of a green pipeline. They want a self-contained, tamper-evident bundle that proves a specific control operated across a specific period: the control narrative, the complete population it acted on, the sample that was tested, the disposition of every exception, and the human sign-offs that closed the loop — all bound together so that nothing can be added, removed, or altered without detection. Evidence packaging is the engineering discipline of turning a reconciliation run and its immutable append-only ledger into exactly that artifact. It sits at the end of the control lifecycle, after matching, exception routing, and resolution have completed, and it converts operational history into a frozen, cryptographically sealed object that an external auditor can open, verify independently, and file as testing evidence. This page is a direct extension of Compliance, Audit Trails & Financial Controls, focused on the mechanics of producing evidence a controls tester will accept without a follow-up request.

Prerequisites: Upstream Control State

The packager never derives facts; it collects and seals facts that already exist. Before a bundle can be built for a reporting period, the following state must be durable and queryable:

• A complete, immutable audit ledger for the period. Every match decision, exception disposition, and sign-off must already be persisted as append-only events in the hash-chained store described in Immutable Append-Only Ledger Design. The packager reads this ledger; it does not write to the control record.
• A closed population boundary. The set of reconciliation items in scope for the period must be finalised — the accounting period is closed and no late postings are outstanding — so the population count is stable and the completeness assertion holds.
• Recorded reviewer sign-offs. Every exception that required human judgement must carry an approver identity and timestamp from the manual review queue, because a sign-off is itself evidence that segregation of duties operated.
• A control narrative and a stable control identifier. The narrative that describes what the control does, its frequency, and its owner must be versioned so the packaged bundle references the exact narrative in force during the period.

If any of these is absent the packager must refuse to emit a bundle rather than produce a partial one, because an evidence bundle that silently omits part of the population is worse than no bundle at all — it misrepresents the completeness assertion an auditor relies on.

What Constitutes Evidence: The Anatomy of a Bundle

SOX §404 controls testing turns on two assertions the auditor makes about your evidence: completeness (the population is the whole population, nothing was excluded) and accuracy (each item faithfully reflects what the system actually did). Every design decision below exists to make those two assertions independently verifiable. A defensible bundle is composed of five distinct evidence classes, each answering a question the tester will ask:

1. Control narrative — what the control is, who owns it, how often it runs, and the assertion it supports. This is the frame; without it the raw data is uninterpretable.
2. Population — the complete, enumerated set of reconciliation items the control acted on during the period, with a count and a boundary definition (period start, period end, ledger entities in scope). Completeness testing lives here.
3. Sample selection — the subset the auditor (or the control’s own testing procedure) examines in detail, together with the method that produced it. The method must be reproducible: a different engineer, given the same population and the same seed, must select the identical sample.
4. Exception disposition — for every break that surfaced, its classification, its resolution code, the adjustment amount in Decimal, and the reviewer who signed it off. This proves the control not only detected but resolved.
5. Sign-offs — the segregation-of-duties evidence: who approved what, when, and that no approver approved their own adjustment.

Binding these together is a manifest: a machine-readable index that names every file in the bundle and records the sha256 digest of its exact bytes. The manifest is then itself hashed and signed, and the whole set is written into a single sealed archive. The signature over the manifest is what makes the bundle tamper-evident — change any byte of any evidence file and its digest no longer matches the manifest; change the manifest and its signature no longer verifies.

Deterministic Sample Selection & the Evidence Manifest

The single most common reason an evidence bundle fails audit review is that the sample cannot be reproduced. If the sampling procedure uses an unseeded random source, the wall clock, or a set-iteration order that varies between runs, then re-running the packager produces a different sample and the auditor can no longer confirm that the tested sample was drawn honestly from the stated population. The fix is a seeded, deterministic selector: sort the population by a stable key, then draw with a pseudo-random generator initialised from a recorded seed. The seed is stored in the manifest, so the selection is a pure function of (population, seed, rate) and is reproducible forever.

The manifest is the spine of the bundle. It enumerates each evidence file, records the sha256 of its canonical bytes, and captures the metadata needed to re-derive the sample: the population digest, the seed, the sampling rate, and the period boundary. Hashing the canonical serialisation matters — the same logical content must always produce the same bytes, which means sorted keys, a fixed separator, and UTF-8 encoding, so the digest is stable across machines and Python versions.

Once the manifest is assembled it is serialised canonically, its own digest computed, and that digest signed. Chain-of-custody is then a property of the archive: the sealed bundle records who built it, when, from which ledger snapshot, and under which signing key, so custody can be traced from the reconciliation run through the packager to the auditor who opens it. The pipeline below shows the flow from ledger to auditor:

The complexity of the whole build is O(n log n) in the population size n, dominated by the canonical sort used for both deterministic selection and completeness enumeration; hashing is linear in total evidence bytes. The dominant caveat is immutability: once the manifest is signed, the bundle is frozen. Any correction — a late adjustment, a re-classified exception — must produce a new bundle with a new signature, never an in-place edit, because editing a sealed bundle destroys exactly the tamper-evidence property that makes it worth producing.

Production-Grade Python Implementation

The packager below collects audit events for a period, computes a sha256 manifest over the canonical bytes of every evidence file, selects a reproducible sample from a recorded seed, signs the manifest, and seals everything into a single archive. Every packaging action emits a structured audit record carrying the trace_id, the source_hash, and the match_decision, so the act of building evidence is itself an auditable event. Monetary values are Decimal throughout and never touch float.

import hashlib
import hmac
import io
import json
import logging
import tarfile
import uuid
from dataclasses import dataclass
from datetime import datetime, timezone
from decimal import Decimal
from random import Random
from typing import Iterable

logger = logging.getLogger("compliance.evidence_packager")


def canonical_bytes(obj: object) -> bytes:
    # Stable serialisation: sorted keys, fixed separators, UTF-8.
    # The same logical content must always yield the same bytes.
    return json.dumps(
        obj, sort_keys=True, separators=(",", ":"), default=str
    ).encode("utf-8")


def sha256_hex(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()


@dataclass(frozen=True)
class AuditEvent:
    event_id: str
    reconciliation_id: str
    match_decision: str          # MATCHED | PARTIAL | UNMATCHED
    disposition: str             # AUTO_CLEARED | REVIEWED | ESCALATED
    adjustment_amount: Decimal   # exact; never float
    approver: str | None
    occurred_at: datetime        # timezone-aware UTC


def select_sample(
    population: list[AuditEvent], rate: Decimal, seed: int
) -> list[AuditEvent]:
    """Deterministic, reproducible selection.

    A pure function of (population, rate, seed): the same inputs always
    return the identical sample, so an auditor can reproduce it exactly.
    """
    if not population:
        return []
    # Sort by a stable key so iteration order can never vary between runs.
    ordered = sorted(population, key=lambda e: e.event_id)
    k = max(1, int((Decimal(len(ordered)) * rate).to_integral_value()))
    rng = Random(seed)  # seeded PRNG; no wall clock, no os.urandom
    idx = sorted(rng.sample(range(len(ordered)), k))
    return [ordered[i] for i in idx]


def build_manifest(
    control_id: str,
    period: str,
    seed: int,
    rate: Decimal,
    files: dict[str, bytes],
    population_digest: str,
) -> dict:
    entries = [
        {"name": name, "sha256": sha256_hex(body), "bytes": len(body)}
        for name, body in sorted(files.items())
    ]
    return {
        "control_id": control_id,
        "period": period,
        "population_digest": population_digest,
        "sample_seed": seed,
        "sample_rate": str(rate),
        "built_at": datetime.now(timezone.utc).isoformat(),
        "files": entries,
    }


def sign_manifest(manifest: dict, signing_key: bytes) -> str:
    # HMAC-SHA256 over the canonical manifest bytes. In production a
    # KMS-held asymmetric key is preferred; the digest-then-sign shape
    # is identical either way.
    payload = canonical_bytes(manifest)
    return hmac.new(signing_key, payload, hashlib.sha256).hexdigest()


def package_evidence(
    control_id: str,
    period: str,
    events: Iterable[AuditEvent],
    signing_key: bytes,
    key_ref: str,
    rate: Decimal = Decimal("0.10"),
    seed: int = 20260715,
) -> bytes:
    population = list(events)
    if not population:
        raise ValueError("INCOMPLETE_POPULATION: empty population for period")

    trace_id = str(uuid.uuid4())
    pop_records = [
        {
            "event_id": e.event_id,
            "reconciliation_id": e.reconciliation_id,
            "match_decision": e.match_decision,
            "disposition": e.disposition,
            "adjustment_amount": str(e.adjustment_amount),
            "approver": e.approver,
            "occurred_at": e.occurred_at.astimezone(timezone.utc).isoformat(),
        }
        for e in population
    ]
    population_bytes = canonical_bytes(pop_records)
    population_digest = sha256_hex(population_bytes)

    sample = select_sample(population, rate, seed)
    sample_bytes = canonical_bytes([e.event_id for e in sample])

    files: dict[str, bytes] = {
        "population.json": population_bytes,
        "sample.json": sample_bytes,
    }
    manifest = build_manifest(
        control_id, period, seed, rate, files, population_digest
    )
    manifest_bytes = canonical_bytes(manifest)
    signature = sign_manifest(manifest, signing_key)

    seal = {
        "manifest_sha256": sha256_hex(manifest_bytes),
        "signature": signature,
        "signing_key_ref": key_ref,
        "algorithm": "HMAC-SHA256",
        "trace_id": trace_id,
    }

    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode="w:gz") as tar:
        for name, body in {
            **files,
            "manifest.json": manifest_bytes,
            "seal.json": canonical_bytes(seal),
        }.items():
            info = tarfile.TarInfo(name=name)
            info.size = len(body)
            info.mtime = 0  # fixed mtime keeps the archive byte-reproducible
            tar.addfile(info, io.BytesIO(body))

    logger.info(
        "evidence_bundle_sealed",
        extra={
            "trace_id": trace_id,
            "source_hash": population_digest,
            "match_decision": "PACKAGED",
            "control_id": control_id,
            "period": period,
            "population_count": len(population),
            "sample_count": len(sample),
            "manifest_sha256": seal["manifest_sha256"],
            "signing_key_ref": key_ref,
            "ts": datetime.now(timezone.utc).isoformat(),
        },
    )
    return buf.getvalue()

The seal binds the manifest digest, the signature, and the key reference together, so verification is a three-step check anyone can run offline: recompute each file’s sha256 and confirm it matches the manifest, recompute the manifest digest and confirm it matches the seal, then verify the signature over the manifest bytes. Because select_sample is a pure function of (population, rate, seed) and the archive uses a fixed mtime, two independent runs from the same ledger snapshot produce byte-identical bundles — the strongest possible form of the accuracy assertion.

Configuration Reference

Every packaging parameter is version-controlled and captured in the manifest, so an auditor can reconstruct exactly how a given bundle was produced. Treat the defaults as a conservative baseline and change them only with a recorded rationale, since altering the sampling rate or seed changes which items were tested.

Multi-Dimensional Validation

A bundle is only defensible if it survives independent verification along several axes at once; a single digest check is not enough. The packager validates four orthogonal properties before it is considered sealed, and a controls tester re-runs the same checks on receipt:

1. File integrity — every file’s recomputed sha256 equals its manifest entry. A mismatch means the bundle was altered after sealing and raises MANIFEST_HASH_MISMATCH.
2. Population completeness — the population count and boundary in the manifest match the closed period’s ledger extent; a gap raises INCOMPLETE_POPULATION.
3. Sample reproducibility — re-running select_sample with the manifest’s recorded seed and rate reproduces the sample byte-for-byte; divergence raises SAMPLE_NONREPRODUCIBLE.
4. Signature validity — the seal’s signature verifies against the manifest bytes under the referenced key; an unsigned or unverifiable bundle raises UNSIGNED_EVIDENCE.

def verify_bundle(files: dict[str, bytes], manifest: dict,
                  seal: dict, signing_key: bytes) -> None:
    for entry in manifest["files"]:
        body = files[entry["name"]]
        if sha256_hex(body) != entry["sha256"]:
            raise ValueError(f"MANIFEST_HASH_MISMATCH: {entry['name']}")
    if sha256_hex(canonical_bytes(manifest)) != seal["manifest_sha256"]:
        raise ValueError("MANIFEST_HASH_MISMATCH: manifest digest")
    expected = sign_manifest(manifest, signing_key)
    if not hmac.compare_digest(expected, seal["signature"]):
        raise ValueError("UNSIGNED_EVIDENCE: signature does not verify")

Because the checks are independent, a partial pass is diagnostic: a bundle whose files verify but whose signature does not tells you the manifest is intact but the seal was reissued under the wrong key, which is a different remediation from a content mismatch.

Async / Batch Packaging Patterns

At quarter-end a single controls program may need to package hundreds of bundles — one per control, per entity, per period. Because each bundle is an independent, CPU-bound sealing job (canonical hashing plus gzip), the throughput pattern is a bounded asyncio fan-out that dispatches each build to a worker thread so the event loop is never blocked on hashing:

import asyncio


async def package_all(
    jobs: list[dict], signing_key: bytes, concurrency: int = 4
) -> list[bytes]:
    sem = asyncio.Semaphore(concurrency)

    async def one(job: dict) -> bytes:
        async with sem:  # bound concurrency to protect the signing service
            return await asyncio.to_thread(
                package_evidence,
                job["control_id"], job["period"], job["events"],
                signing_key, job["key_ref"],
            )

    return await asyncio.gather(*(one(j) for j in jobs))

The semaphore bounds concurrency so a burst of quarter-end jobs cannot exhaust the signing service’s rate limit, and asyncio.to_thread keeps each CPU-bound seal off the event loop. Bundles are written to a WORM (Write Once, Read Many) object store keyed by control identifier and period, so a re-run for the same control never overwrites a prior sealed bundle — it lands beside it under a new object version, preserving the full custody history.

Failure Modes & Remediation

Each named code is emitted with the trace_id and source_hash so it can be triaged directly from the audit stream. The unifying principle is that the packager never emits a bundle it could not fully seal and verify.

When a bundle covers exceptions that required human judgement, the sign-offs it packages originate in the manual review queue: each approver identity and timestamp the queue recorded becomes segregation-of-duties evidence inside exception disposition, which is why a missing sign-off surfaces as a control deficiency rather than a mere data gap.

Compliance & Audit Trail Requirements

SOX §404 requires management to test that internal controls over financial reporting operated effectively, and the evidence bundle is the artifact that survives external audit. Three properties make it defensible:

1. Tamper-evidence through signing. The manifest binds a sha256 of every evidence file; the seal binds a digest of the manifest and a signature over it. Any post-hoc modification breaks at least one link in the chain, so the auditor can trust that what they open is what was sealed.
2. Reproducible sampling. Because selection is a pure function of the recorded (population, rate, seed), the auditor can reproduce the tested sample independently, which is the practical test of the accuracy assertion. Selection methodology that cannot be reproduced is not evidence.
3. Chain-of-custody and retention. Each bundle records who built it, when, from which ledger snapshot, and under which key reference, and is retained in a WORM store for the mandated period (typically seven years). Custody is traceable from the reconciliation run through the packager to the auditor.

Where these bundles gate a release — proving that controls evidence was produced before a reconciliation change ships — the packaging step is wired into the deploy pipeline described in CI/CD Gating for Financial Controls. A concrete, end-to-end build of one such bundle, including the KMS signing call and the archive layout, is walked through in Generating SOX 404 evidence bundles in Python. By treating evidence packaging as a deterministic, cryptographically sealed transformation of the audit ledger rather than an ad-hoc export, controls and accounting-engineering teams produce evidence an external auditor accepts on first read.

Related

• Immutable Append-Only Ledger Design
• CI/CD Gating for Financial Controls
• Generating SOX 404 Evidence Bundles in Python
• Manual Review Queue Design

Part of Compliance, Audit Trails & Financial Controls.