Error Categorization & Retry Logic: Turning Portal Failures Into Deterministic State Transitions
This guide is part of the Secretary of State Portal & API Ingestion framework, and it owns one job in that stack: deciding what a pipeline does the instant a state portal returns something other than a clean success. A filing run touches dozens of jurisdictions with incompatible failure semantics, and a single mishandled exception can cascade into a late fee, a duplicate franchise-tax charge, or administrative dissolution. The difference between a resilient pipeline and a liability is whether every error maps to exactly one deterministic next state.
Naive try/except is not enough here. Conflating recoverable latency with an actionable compliance failure corrupts the filing queue and invalidates the audit trail. This layer demands a strict error taxonomy, a policy-driven retry engine that is idempotent and bounded, and a fallback router that preserves chain-of-custody. Legal operations teams depend on it to maintain continuous good standing; Python automation engineers implement it as the stateful boundary that every portal call passes through.
Statutory and Regulatory Context
Retry behavior is not a purely technical concern — it is constrained by statute. Most annual-report and franchise-tax obligations carry a hard due date plus a defined grace window, and the retry engine must guarantee that a transient failure is exhausted before that window closes, not after. Delaware franchise tax and annual reports arise under DGCL § 503 with a March 1 due date; California’s Statement of Information (Cal. Corp. Code § 1502) is anchored to a formation-anniversary window. A retry policy that blindly sleeps for an HTTP-date Retry-After of several days can silently push a submission past a statutory deadline — the policy must clamp delays against the remaining deadline budget supplied by the State Filing Deadline Calendars layer.
Two transport standards govern the mechanics. Backoff and the Retry-After header follow RFC 9110 §10.2.3, where the value is either an integer count of seconds or an HTTP-date string — a distinction the parser must handle without raising. Idempotent retry semantics for non-safe methods follow RFC 9110 §9.2.2: only operations that are idempotent by definition (or made so with an idempotency key) may be retried automatically. Every classification and routing decision is also a record-retention artifact — SOX internal-control and jurisdictional e-filing mandates require that the reason a filing was retried, halted, or escalated be reconstructable under audit.
Architecture and Design Model
The engine sits at a deliberate choke point: every response from the async polling and rate limiting layer is handed to a classifier before any side effect, the classifier emits a RoutingDecision, and only that decision determines whether the pipeline retries, halts, escalates, or falls back. The design rests on three decisions.
Classify before you act. A portal response is never interpreted ad hoc at the call site. It is reduced to one of four categories by a pure, side-effect-free function. This keeps the portal-facing code free of defensive branching and makes every routing outcome unit-testable in isolation.
Retry only what is safe to repeat. Automatic retries apply exclusively to safe, idempotent operations — status checks, good-standing lookups, read polls. Non-idempotent actions (form submission, payment, document upload) are never blindly retried; they are guarded with idempotency keys and acknowledgment tokens so that a repeat is either deduplicated by the portal or routed to fallback.
Exhaustion is a state, not an exception. When retries run out or a permanent failure occurs, the payload transitions to a deterministic fallback path — a dead-letter queue with full request/response capture — rather than disappearing into a swallowed exception. Silent failure is the one outcome the taxonomy forbids.
State portal interactions must be classified into a strict taxonomy that dictates routing behavior, escalation thresholds, and statutory impact. The following matrix governs all exception routing:
| Category | Indicators | Routing action | Compliance impact |
|---|---|---|---|
| Transient network | HTTP 5xx, ETIMEDOUT, TLS renegotiation, DNS resolution failure |
Automatic retry with bounded exponential backoff | None (if resolved within the statutory window) |
| Permanent validation | HTTP 4xx, malformed entity IDs, missing officer signatures, fee mismatches, invalid jurisdiction codes |
Immediate halt, route to legal-ops queue, preserve payload | High (manual correction required before resubmission) |
| Regulatory / admin block | Entity status INACTIVE, franchise-tax holds, scheduled maintenance windows, filing moratoriums |
Trigger penalty-avoidance logic, escalate to compliance dashboard | Critical (requires statutory exemption or deadline extension) |
| Infrastructure drift | Unexpected HTML where JSON was expected, missing DOM selectors, API schema-version mismatch | Graceful degradation, flag for schema-remediation pipeline | Medium (engineering intervention before next cycle) |
Proper classification ensures the ingestion layer never retries a validation failure or silently drops a regulatory block. Every exception maps to a deterministic state transition.
Prerequisites and Dependencies
| Requirement | Version / detail | Rationale |
|---|---|---|
| Python | 3.10+ | X | Y union syntax, structural pattern matching, and match routing used throughout |
| HTTP client | httpx or aiohttp |
Native async support and explicit timeout/connection-reset surfacing |
| Structured logging | stdlib logging with a JSON formatter |
Audit trails must be machine-parseable, not free text |
| Upstream pacing | Async polling and rate-limiting service | Supplies the per-jurisdiction token budget the retry engine must respect |
| Deadline source | State Filing Deadline Calendars | Supplies the remaining-window budget used to clamp Retry-After delays |
| Durable queue | Redis / SQS / Celery broker | Backs the dead-letter and escalation queues that receive exhausted attempts |
The classifier itself is intentionally stateless and performs no network I/O — portal calls, pacing, and retries belong to the surrounding ingestion layer. That separation keeps classification idempotent on replay and trivially testable against captured response fixtures.
Step-by-Step Implementation
Phase 1 — Model the response and the routing decision
Both the inbound response and the outbound decision are frozen dataclasses. Immutability matters here: a RoutingDecision becomes part of the audit record, so it must not be mutable after the classifier returns it.
from __future__ import annotations
import enum
import logging
import random
import time
from dataclasses import dataclass, field
from typing import Any, Callable
logger = logging.getLogger("ingestion.retry_engine")
class ComplianceErrorCategory(str, enum.Enum):
TRANSIENT = "transient"
VALIDATION = "validation"
REGULATORY = "regulatory"
INFRASTRUCTURE = "infrastructure"
@dataclass(frozen=True)
class ComplianceResponse:
status_code: int
headers: dict[str, str]
body: str | None
is_success: bool
@dataclass(frozen=True)
class RoutingDecision:
category: ComplianceErrorCategory
should_retry: bool
retry_delay: float
escalation_required: bool
audit_payload: dict[str, Any] = field(default_factory=dict)
Phase 2 — Classify every response deterministically
The classifier is a pure function: same response in, same decision out, no side effects. It checks categories in priority order so that a regulatory block buried inside a 200 body is never mistaken for success.
def classify_error(response: ComplianceResponse) -> RoutingDecision:
"""Deterministic error categorization with statutory routing logic."""
status = response.status_code
body = (response.body or "").lower()
headers = response.headers
# Transient: 5xx, timeouts, connection resets -> safe to retry
if 500 <= status <= 599:
return RoutingDecision(
category=ComplianceErrorCategory.TRANSIENT,
should_retry=True,
retry_delay=0.0, # actual delay computed by the retry wrapper
escalation_required=False,
audit_payload={"status": status, "reason": "server_error"},
)
# Permanent validation: 4xx or known rejection phrases -> halt, escalate
validation_markers = ("invalid entity", "missing signature", "fee mismatch")
if 400 <= status <= 499 or any(m in body for m in validation_markers):
return RoutingDecision(
category=ComplianceErrorCategory.VALIDATION,
should_retry=False,
retry_delay=0.0,
escalation_required=True,
audit_payload={"status": status, "legal_review_required": True},
)
# Regulatory / admin block -> trigger penalty-avoidance, escalate
regulatory_markers = (
"inactive", "franchise tax hold", "maintenance window", "filing moratorium",
)
if any(m in body for m in regulatory_markers):
return RoutingDecision(
category=ComplianceErrorCategory.REGULATORY,
should_retry=False,
retry_delay=0.0,
escalation_required=True,
audit_payload={"status": status, "penalty_avoidance_triggered": True},
)
# Infrastructure drift: an HTML document or missing schema header on a 200
if status == 200 and ("<!doctype html" in body or "schema_version" not in headers):
return RoutingDecision(
category=ComplianceErrorCategory.INFRASTRUCTURE,
should_retry=False,
retry_delay=0.0,
escalation_required=False,
audit_payload={"status": status, "schema_remediation_required": True},
)
# Unclassified non-success: treat conservatively as transient, retry once
return RoutingDecision(
category=ComplianceErrorCategory.TRANSIENT,
should_retry=True,
retry_delay=0.0,
escalation_required=False,
audit_payload={"status": status, "reason": "unclassified"},
)
The infrastructure-drift check tests for "<!doctype html" in body rather than a vague phrase like "unexpected html" — the former detects an actual HTML document returned where JSON was expected, which is the common legacy-portal failure mode and the exact signal the headless browser fallback strategies layer consumes to switch transports.
Phase 3 — Retry with bounded, RFC-compliant backoff
The wrapper applies full-jitter exponential backoff, honors a server-supplied Retry-After, and clamps every delay against a hard ceiling so a long server-specified wait can never starve the workflow past its deadline budget.
def parse_retry_after(value: str | None) -> float | None:
"""RFC 9110 §10.2.3: Retry-After is delta-seconds OR an HTTP-date."""
if not value:
return None
try:
return max(0.0, float(value)) # integer seconds form
except ValueError:
# HTTP-date form: parse to an absolute instant, return seconds-from-now
from email.utils import parsedate_to_datetime
try:
target = parsedate_to_datetime(value)
except (TypeError, ValueError):
return None
return max(0.0, target.timestamp() - time.time())
def execute_with_retry(
operation: Callable[[], ComplianceResponse],
*,
is_idempotent: bool,
max_retries: int = 4,
base_delay: float = 1.5,
max_delay: float = 60.0,
) -> ComplianceResponse:
"""Bounded retry wrapper; retries only safe, idempotent operations."""
for attempt in range(max_retries + 1):
response = operation()
decision = classify_error(response)
if response.is_success:
return response
# Never auto-retry a non-idempotent action (submission, payment, upload)
if decision.should_retry and not is_idempotent:
logger.warning(
"non_idempotent_no_retry",
extra={"category": decision.category.value, "status": response.status_code},
)
return response
if not decision.should_retry or attempt == max_retries:
logger.warning(
"retry_terminal",
extra={
"category": decision.category.value,
"attempts": attempt,
"escalation_required": decision.escalation_required,
},
)
return response
server_delay = parse_retry_after(response.headers.get("Retry-After"))
backoff = base_delay * (2 ** attempt) + random.uniform(0, 1)
delay = min(server_delay if server_delay is not None else backoff, max_delay)
logger.info(
"retry_scheduled",
extra={"attempt": attempt + 1, "max": max_retries, "delay_s": round(delay, 2)},
)
time.sleep(delay)
raise RuntimeError("unreachable retry state")
Phase 4 — Guard non-idempotent filing actions
Form submission, payment, and document upload are inherently non-idempotent: a blind repeat risks a duplicate filing or a double-charged franchise tax. Instead of retrying the request, the engine retries only a safe status check keyed by an idempotency token, and treats a missing acknowledgment as a fallback trigger rather than a reason to resubmit.
import uuid
def submit_filing_once(
submit: Callable[[str], ComplianceResponse],
verify: Callable[[str], ComplianceResponse],
idempotency_key: str | None = None,
) -> ComplianceResponse:
"""Submit a non-idempotent filing exactly once, then verify by key."""
key = idempotency_key or str(uuid.uuid4())
# The submission carries X-Idempotency-Key; the portal dedupes a replay.
response = submit(key)
decision = classify_error(response)
# A successful submission MUST return an acknowledgment token.
if response.is_success and _has_ack_token(response):
return response
# No token: do NOT resubmit. Verify status (idempotent) before any fallback.
if decision.category is ComplianceErrorCategory.TRANSIENT:
status_check = execute_with_retry(lambda: verify(key), is_idempotent=True)
if status_check.is_success and _has_ack_token(status_check):
return status_check
logger.error("filing_unconfirmed", extra={"idempotency_key": key})
return response # caller routes to dead-letter / escalation
def _has_ack_token(response: ComplianceResponse) -> bool:
body = (response.body or "").lower()
return "submission_id" in body or "transaction_ref" in body
Phase 5 — Route exhausted attempts to auditable fallback
When retries are exhausted or a permanent failure occurs, the workflow serializes the full context into an immutable record and routes by category. Regulatory blocks additionally trigger penalty-avoidance logic that calculates the statutory grace window — the same mechanism documented in calculating penalty risk scores based on state grace periods.
def route_to_fallback(
response: ComplianceResponse,
decision: RoutingDecision,
*,
entity_id: str,
jurisdiction: str,
enqueue: Callable[[str, dict[str, Any]], None],
) -> None:
"""Serialize the failure and route it to a durable queue by category."""
record = {
"entity_id": entity_id,
"jurisdiction": jurisdiction,
"category": decision.category.value,
"status_code": response.status_code,
"headers": dict(response.headers),
"body": response.body,
"audit": decision.audit_payload,
"ts": time.time(),
}
queue = {
ComplianceErrorCategory.VALIDATION: "legal_ops_queue",
ComplianceErrorCategory.REGULATORY: "penalty_avoidance_queue",
ComplianceErrorCategory.INFRASTRUCTURE: "schema_remediation_queue",
ComplianceErrorCategory.TRANSIENT: "dead_letter_queue",
}[decision.category]
enqueue(queue, record)
logger.info("fallback_routed", extra={"queue": queue, "entity_id": entity_id})
Edge Cases and Portal-Specific Gotchas
Portal failure semantics are not uniform. The same logical condition surfaces as different status codes, headers, and body shapes across jurisdictions, so the classifier’s marker lists are jurisdiction-aware in practice.
| Jurisdiction | Portal | Gotcha | Required handling |
|---|---|---|---|
| Delaware | Division of Corporations | Returns a 200 with an HTML maintenance page during nightly batch windows |
Treat HTML-on-200 as infrastructure drift, not success; retry after the window |
| California | bizfile Online | Rate-limit responses arrive as 403 with a body phrase, not 429; no Retry-After |
Match the body marker, classify as transient, fall back to computed backoff |
| New York | DOS e-filing | A 5xx may persist for hours during scheduled outages |
Clamp total retry span against the deadline budget; escalate before the window closes |
| Texas | SOSDirect | Returns 200 with a session-expired notice instead of a 401 |
Detect the session marker as validation; re-authenticate rather than retry the call |
Two cross-jurisdiction traps deserve emphasis. A Retry-After expressed as an HTTP-date can specify a delay measured in days — never sleep on it unconditionally; clamp to max_delay and re-evaluate against the deadline budget. And a 200 OK is not proof of success: a portal that wraps an error in an HTML page or omits the acknowledgment token has failed, and the classifier must treat the absence of a positive success signal as a failure.
Verification and Testing
Because the classifier is pure, it is tested against captured response fixtures with no network access. The retry wrapper is tested with an operation stub that returns a scripted sequence of responses, and time.sleep is patched so the suite runs instantly.
import unittest
from unittest.mock import patch
class ClassifyErrorTests(unittest.TestCase):
def test_html_on_200_is_infrastructure_drift(self) -> None:
resp = ComplianceResponse(200, {}, "<!DOCTYPE html><html>...", is_success=False)
decision = classify_error(resp)
self.assertEqual(decision.category, ComplianceErrorCategory.INFRASTRUCTURE)
self.assertFalse(decision.should_retry)
def test_regulatory_block_escalates(self) -> None:
resp = ComplianceResponse(200, {"schema_version": "3"}, "entity INACTIVE", False)
decision = classify_error(resp)
self.assertEqual(decision.category, ComplianceErrorCategory.REGULATORY)
self.assertTrue(decision.escalation_required)
def test_5xx_is_retryable_transient(self) -> None:
decision = classify_error(ComplianceResponse(503, {}, "", is_success=False))
self.assertTrue(decision.should_retry)
class RetryWrapperTests(unittest.TestCase):
def test_non_idempotent_is_never_retried(self) -> None:
calls = {"n": 0}
def op() -> ComplianceResponse:
calls["n"] += 1
return ComplianceResponse(503, {}, "", is_success=False)
with patch("time.sleep"):
execute_with_retry(op, is_idempotent=False, max_retries=4)
self.assertEqual(calls["n"], 1) # called once, no retry
if __name__ == "__main__":
unittest.main()
Integration tests should target each portal’s sandbox endpoint where one exists (California’s bizfile sandbox, Delaware’s test gateway) and assert that a forced maintenance page is classified as infrastructure drift rather than success.
Troubleshooting
Duplicate filings appear for the same entity in one cycle
A non-idempotent submission was retried directly. Confirm the submit path runs through submit_filing_once and that is_idempotent=False is passed for any POST/PUT. The retry of a status check is correct; the retry of the submission itself is the bug. Verify the portal honors X-Idempotency-Key — those that do not require the acknowledgment-token guard instead.
A filing was retried past its statutory deadline
The retry span was not clamped against the remaining deadline budget. The wrapper’s max_delay bounds a single sleep, but the cumulative span must also be checked against the window from the deadline calendar. When a server Retry-After would push the cumulative wait past the window, stop retrying and escalate to penalty-avoidance immediately.
A regulatory block was silently swallowed as a success
The response was a 200 whose body carried a hold or moratorium phrase the marker list did not cover. Add the jurisdiction’s exact phrase to regulatory_markers and add a fixture asserting escalation. Remember that category checks run in priority order — regulatory and validation markers are evaluated before the 200/success path precisely so a wrapped error is never treated as a clean response.
The pipeline hammers a portal and triggers an IP ban
Backoff is firing but pacing is not coordinated with the upstream limiter. Retries must draw from the same per-jurisdiction token budget the async polling and rate-limiting layer enforces; otherwise concurrent retries overlap and breach the portal’s request ceiling. Route every retry through the shared limiter rather than sleeping independently.
Retry-After parsing raises on certain portals
The header arrived in HTTP-date form and a bare float() raised ValueError. Use parse_retry_after, which falls back to email.utils.parsedate_to_datetime and returns seconds-from-now, defaulting to computed backoff when the value is unparseable. Never let header parsing raise into the retry loop.
Operational Checklist
Frequently Asked Questions
How many retries are appropriate for a state portal?
Three to five bounded attempts cover the overwhelming majority of transient 5xx and timeout conditions without risking deadline starvation. The exact number matters less than the two hard constraints around it: a per-attempt max_delay ceiling, and a cumulative span clamped to the remaining statutory window from the deadline calendar. Retrying forever against a portal in a multi-hour outage is worse than escalating early, because escalation preserves the option to file by an alternate path before the deadline.
Why classify a 200 OK as a failure?
Legacy state portals routinely return 200 OK while the body carries a maintenance page, a session-expired notice, or a regulatory hold. Treating the status code as the sole success signal lets these slip through as confirmed filings. The classifier requires a positive success signal — an acknowledgment token, a schema_version header, structured JSON — and treats its absence as a failure to be categorized, not a success.
Can payment and submission endpoints ever be retried automatically?
Not the write itself. A POST that charges a franchise tax or files a report is non-idempotent, and a blind repeat risks a duplicate filing or double charge. What is safe to retry is a status verification keyed by an idempotency token: submit once, then poll the status endpoint by key. Portals that honor X-Idempotency-Key will dedupe a genuine replay; those that do not require the acknowledgment-token guard so a missing token routes to fallback instead of triggering a resubmit.
Where does this layer hand off when a portal is structurally broken?
An infrastructure-drift classification — HTML where JSON was expected, or a missing schema version — is the signal to switch transports. The schema-remediation queue captures it for engineering, and the same condition is what the headless browser fallback strategies layer consumes to retrieve the record through a browser session while preserving the identical error taxonomy and audit standards.