Checkpointing and Resuming Interrupted Filing Batches #

This guide is part of the Multi-Entity Batch Orchestration area within the Deadline Tracking & Routing Engines framework: it addresses what happens when a batch of thousands of filings is halfway through and the worker dies — a deploy, an OOM kill, a lost spot instance — and the run has to start again without re-filing everything it already filed. The parent area defines how filings execute concurrently and idempotently in the happy path; this page makes that execution durable across a crash, so a restart resumes from the last recorded position instead of replaying the whole batch or, worse, submitting a second filing for entities that already succeeded.

Scope of This Page #

This page covers durable checkpointing and resumption: recording each entity’s progress to a durable store as it advances, replaying idempotent steps safely, guaranteeing exactly-once submission when a resumed run re-touches an entity that may have partially completed, and reconciling entities whose state is ambiguous after the interruption. It deliberately excludes the mechanics it builds on. It does not set the per-jurisdiction concurrency ceilings or session handling — that is Orchestrating Parallel Filing Batches Across Multiple Jurisdictions. It does not decide the order in which entities are admitted; that ordering comes from Priority Scoring Algorithms. Here we assume a scored, concurrency-bounded batch is already running and focus only on surviving its interruption.

The Constraint That Makes Exactly-Once Non-Negotiable #

A resumed batch is dangerous precisely where an ordinary program is safe: re-running a step. In compliance filing, the submission step has an external side effect that money and legal status ride on — a duplicate Statement of Information filed twice in California’s 72-hour window is rejected under the portal’s own de-duplication, but a duplicate franchise-tax payment against Delaware under 8 Del. C. § 502 is a real double charge, and a re-submitted annual report can trigger conflicting records that put an entity’s good standing in question. Because the underlying obligations are statutorily independent and non-fungible, the batch cannot be wrapped in a single transaction that rolls back cleanly; each entity’s filing commits on its own portal clock. Durability therefore has to be per-entity and idempotent: every step records a checkpoint the instant it completes, and every re-executed step keys on a stable idempotency token so that a submission which already reached the registry is recognized and skipped rather than repeated. Exactly-once is not a performance nicety here — it is the difference between a clean resume and a duplicate legal filing.

Prerequisites #

• Python 3.10+ — for match, X | Y unions, and modern typing.
• A durable checkpoint store — a write-once/updatable row per entity in Postgres, DynamoDB, or the broker’s own state table — that survives a worker restart.
• A stable idempotency key per (entity, jurisdiction, filing_type, filing_year), identical across the original run and every resume.
• Idempotent step handlers: validation and reconciliation must be pure re-runs, and the submission handler must consult the registry’s idempotency header before sending.
• A reconciliation path for entities left in an ambiguous SUBMITTED-but-unconfirmed state at the moment of the crash.

Implementation: A Checkpointed, Resumable Batch Runner #

The runner below advances each entity through an ordered pipeline of steps and commits a durable checkpoint after every step completes. On startup it loads the checkpoint for each entity and resumes at the first step after the last one recorded, so completed entities are skipped entirely and in-flight ones continue from where they stopped. The submission step is guarded by the entity’s idempotency key: if a checkpoint shows the submission already reached the registry, the runner reconciles the existing filing instead of sending a second one. Comments mark the compliance-critical lines.

from __future__ import annotations

import json
import logging
from dataclasses import dataclass
from datetime import datetime, timezone
from enum import IntEnum
from typing import Protocol

# Structured JSON logging — every checkpoint and resume decision is auditable.
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("orchestration.checkpoint")


class Step(IntEnum):
    # Ordered pipeline; a checkpoint records the highest COMPLETED step.
    QUEUED = 0
    VALIDATED = 1
    SUBMITTED = 2      # external side effect — the exactly-once boundary
    RECONCILED = 3     # terminal success


@dataclass(frozen=True)
class Checkpoint:
    entity_id: str
    idempotency_key: str          # stable across the original run and every resume
    last_step: Step
    updated_at_utc: str


class CheckpointStore(Protocol):
    """Durable store that survives a worker restart."""
    def get(self, entity_id: str) -> Checkpoint | None: ...
    def put(self, checkpoint: Checkpoint) -> None: ...


class RegistryClient(Protocol):
    def already_filed(self, idempotency_key: str) -> bool: ...   # portal-side dedup
    def submit(self, entity_id: str, idempotency_key: str) -> None: ...
    def confirm(self, idempotency_key: str) -> bool: ...


class CheckpointedBatchRunner:
    """Advance entities through the pipeline, checkpointing so a crash is resumable."""

    def __init__(self, store: CheckpointStore, registry: RegistryClient) -> None:
        self._store = store
        self._registry = registry

    def _commit(self, entity_id: str, key: str, step: Step) -> None:
        # Persist BEFORE moving on: the checkpoint is the durable record of truth.
        cp = Checkpoint(entity_id, key, step, datetime.now(timezone.utc).isoformat())
        self._store.put(cp)
        logger.info(json.dumps({
            "event": "checkpoint_committed", "entity_id": entity_id,
            "idempotency_key": key, "last_step": step.name,
        }))

    def run_entity(self, entity_id: str, idempotency_key: str) -> Step:
        cp = self._store.get(entity_id)
        resume_from = cp.last_step if cp else Step.QUEUED
        if resume_from >= Step.RECONCILED:
            # Already terminal on a prior run — skip entirely, do not re-execute.
            logger.info(json.dumps({
                "event": "entity_skipped_completed", "entity_id": entity_id,
                "idempotency_key": idempotency_key,
            }))
            return Step.RECONCILED

        logger.info(json.dumps({
            "event": "entity_resumed", "entity_id": entity_id,
            "resume_from": resume_from.name, "idempotency_key": idempotency_key,
        }))

        # VALIDATE: pure re-run, always safe to repeat.
        if resume_from < Step.VALIDATED:
            self._validate(entity_id)
            self._commit(entity_id, idempotency_key, Step.VALIDATED)

        # SUBMIT: the one step with an external side effect. On resume we must not
        # send a second filing, so consult the registry's idempotency guard first.
        if resume_from < Step.SUBMITTED:
            if self._registry.already_filed(idempotency_key):
                # The pre-crash submission reached the portal; do NOT re-submit.
                logger.warning(json.dumps({
                    "event": "submit_deduplicated", "entity_id": entity_id,
                    "idempotency_key": idempotency_key,
                }))
            else:
                self._registry.submit(entity_id, idempotency_key)
            self._commit(entity_id, idempotency_key, Step.SUBMITTED)

        # RECONCILE: verify the registry actually confirmed before marking terminal.
        if not self._registry.confirm(idempotency_key):
            logger.error(json.dumps({
                "event": "reconcile_unconfirmed", "entity_id": entity_id,
                "idempotency_key": idempotency_key,
            }))
            return Step.SUBMITTED  # ambiguous — leave for the reconciliation path
        self._commit(entity_id, idempotency_key, Step.RECONCILED)
        return Step.RECONCILED

    def run_batch(self, entities: list[tuple[str, str]]) -> dict[str, Step]:
        results: dict[str, Step] = {}
        for entity_id, key in entities:
            try:
                results[entity_id] = self.run_entity(entity_id, key)
            except Exception as exc:  # isolate one entity's failure from the batch
                logger.error(json.dumps({
                    "event": "entity_failed", "entity_id": entity_id,
                    "error_type": type(exc).__name__,
                }))
                results[entity_id] = self._store.get(entity_id).last_step \
                    if self._store.get(entity_id) else Step.QUEUED
        return results

    def _validate(self, entity_id: str) -> None:
        # Placeholder for schema/statutory validation; pure and repeatable.
        logger.info(json.dumps({"event": "validated", "entity_id": entity_id}))

The runner treats the checkpoint store as the single source of truth about progress: a step is only “done” once its checkpoint is durably written, so a crash between doing the work and committing the checkpoint simply re-runs that step on resume — which is safe for the pure steps and idempotency-guarded for the one step that is not. The subtle case is SUBMITTED-but-unconfirmed: the entity submitted, crashed before reconciling, and on resume the registry’s already_filed check confirms the filing exists, so the runner reconciles it rather than filing again.

Configuration Reference #

These parameters govern durability and the exactly-once boundary, not throughput; throughput ceilings live in the parent area.

Failure Modes and Fallback Routing #

Each fault maps onto the parent area’s four-tier taxonomy — transient, statutory, data-validation, and system — and the runner responds to each differently.

1. Crash between the side effect and the checkpoint (system). The entity submitted to the registry but the process died before _commit(..., SUBMITTED). On resume the checkpoint still shows VALIDATED, so the runner re-enters SUBMIT — but already_filed(idempotency_key) returns true, the second submission is suppressed, and the checkpoint advances. This is exactly the window the idempotency guard exists to close.
2. SUBMITTED-but-unconfirmed at the moment of the crash (transient / ambiguous). The filing reached the portal but reconciliation never ran. The runner leaves the entity at SUBMITTED — not terminal — and the reconciliation path re-queries the registry; only a confirmed acknowledgement and matched fee advance it to RECONCILED, mirroring the reconciliation discipline of the parent orchestrator.
3. A poison entity fails deterministically on every resume (data-validation). An entity with a malformed payload crashes the worker each time it is reached, and a naive resume replays the whole batch straight back into it. Isolate per-entity failures (the run_batch try/except), cap the resume count per entity, and quarantine a repeatedly failing entity to the dead-letter queue with its checkpoint attached instead of letting it stall the batch.
4. A stale idempotency key after a re-scored batch (statutory / correctness). If the batch is re-scored and re-admitted through Priority Scoring Algorithms with a different key derivation, the guard no longer recognizes the pre-crash submission and double-files. The key must be a pure function of the obligation identity — never of run-time ordering or attempt count — so it is stable across scoring passes and resumes alike.

Frequently Asked Questions #

Related #

• Multi-Entity Batch Orchestration
• Orchestrating Parallel Filing Batches Across Multiple Jurisdictions
• Priority Scoring Algorithms
• Deadline Tracking & Routing Engines