Secretary Of State Portal Api Ingestion

Polling Secretary of State Portals Without Tripping Rate Limits or Missing Deadlines

This guide is part of the Secretary of State Portal & API Ingestion framework — it owns the pacing and concurrency layer that every bulk status sweep, good-standing check, and annual-report submission runs through. The engineering problem is narrow but unforgiving: state portals enforce undocumented, jurisdiction-specific rate ceilings, and the same requests that breach those ceilings are the ones carrying statutory deadlines. Poll too aggressively and a portal blacklists your egress IPs mid-sweep; poll too cautiously and a delinquent entity slips past its cure window. Async polling and rate limiting is where those two failure modes are reconciled into a single, deterministic execution model.

Statutory and Regulatory Context

The reason a 429 from a state portal is a compliance event — not just a network nuisance — is that the request behind it is usually a statutory obligation with a hard clock. Delaware franchise-tax and annual-report deadlines fall on March 1 for corporations under 8 Del. C. § 502, and a missed filing accrues a $200 penalty plus 1.5% monthly interest before administrative action under § 510. California’s biennial Statement of Information is governed by Cal. Corp. Code § 1502, with a $250 penalty under § 2204 and suspension of corporate powers under § 2205 for non-filers. New York requires a Biennial Statement under N.Y. BCL § 408, and Texas ties franchise-tax forfeiture of the right to transact business to Tex. Tax Code § 171.251.

Because these penalties trigger on the filing clock and not the polling clock, the ingestion layer cannot treat a rate-limited request as a soft failure to be retried whenever convenient. A poller that blindly backs off on a 429 can push a deadline-critical status check past the entity’s grace period — converting an infrastructure constraint into a statutory penalty. Every pacing decision in this subsystem is therefore made against the remaining compliance window, and every attempt is recorded for the same audit standard (NIST SP 800-92 log integrity) that the rest of Secretary of State Portal & API Ingestion applies. The grace-period inputs themselves come from the State Filing Deadline Calendars layer, so the poller reads statutory deadlines rather than guessing them.

Architecture and Design Model

The subsystem resolves every polling operation to a single, auditable outcome. The core design decisions are:

  • Single-intent idempotency. Each unit of work is an immutable ComplianceTask keyed by filing_id. Concurrent requests for the same filing collapse onto one in-flight coroutine, so a state response maps to exactly one compliance record — no duplicate status evaluations during a high-volume renewal window.
  • Per-jurisdiction pacing, not global. A single global rate limit is wrong in both directions: too slow for permissive portals, too fast for fragile ones. Each jurisdiction gets its own token bucket calibrated to that portal’s observed ceiling.
  • Deadline-aware backoff. Standard exponential backoff is overridden when the statutory deadline is near. Urgent tasks short-circuit onto a low-concurrency priority channel rather than queuing behind a bulk sweep.
  • Deterministic error classification. Portal failures are sorted into a fixed taxonomy (rate-limited, transient, portal-blocked, schema-mismatch, permanent) so retry budget is only ever spent on recoverable states.
One ComplianceTask: token-paced acquire, request, classify, and five deterministic outcomes A per-jurisdiction token bucket gates the Acquire step; the task requests entity status and classifies the response. A 200 with a valid body resolves ACTIVE, a 429 takes deadline-aware backoff and a 5xx or schema error takes a short transient wait — both loop back to Acquire — while a blocked portal routes to headless fallback (PENDING_REVIEW) and exhausted attempts resolve to DELINQUENT. one ComplianceTask — token-paced acquire → request → classify → resolve Token bucket per jurisdiction capacity 3 · 1 token / request refill_rate calibrated per portal DE .33 · CA .25 · TX .40 tok/s continuous monotonic refill Acquire token cooperative wait until available Request status GET api.{j}.sos.gov/…/status classify_error status code + body text → category ↻ retry: backoff & transient re-enter Acquire 200 ACTIVE — resolve requires non-empty, schema-valid body 429 RATE_LIMITED → deadline-aware backoff ↻ urgent ≤ 48h: fixed 5s on priority channel else: sleep min(2ⁿ, 60s) + full jitter 5xx / 400 schema TRANSIENT — sleep 2s, retry ↻ captcha / blocked PORTAL_BLOCKED → PENDING_REVIEW hand off to headless fallback, stop retrying attempt > max DELINQUENT attempts exhausted — emit compliance alert

Prerequisites and Dependencies

Component Requirement Rationale
Python 3.10+ Structural pattern matching and X | Y union syntax used throughout the poller
aiohttp ≥ 3.9 Cooperative async HTTP with connection pooling and per-request timeouts
asyncio stdlib Task scheduling, single-intent coroutine deduplication, cooperative sleeps
Egress Rotating/static pool, per-jurisdiction Isolates a blacklisted IP to one portal instead of the whole sweep
Clock source UTC, NTP-synced Deadline math must be timezone-correct; all timestamps stored ISO 8601 / UTC
Compliance ledger Append-only store (e.g. Postgres + WORM, or object storage) Immutable audit trail of every attempt and resolution

Infrastructure assumption: the poller runs as a worker behind a queue, not in request/response context. Deadline metadata is supplied by the calendar layer; egress identity is supplied per jurisdiction so one portal’s defenses never throttle another.

Step-by-Step Implementation

Phase 1 — Model the task and the token bucket

Work is represented as a frozen ComplianceTask so it is safe to pass between workers and to use as an idempotency key. Pacing state lives in a mutable TokenBucket that refills continuously against a monotonic clock.

import asyncio
import logging
import time
from dataclasses import dataclass, field
from datetime import datetime, timedelta, timezone
from enum import Enum, auto

logger = logging.getLogger("compliance.poller")


class ComplianceStatus(Enum):
    ACTIVE = auto()
    DELINQUENT = auto()
    DISSOLVED = auto()
    PENDING_REVIEW = auto()


class ComplianceErrorCategory(Enum):
    TRANSIENT_NETWORK = auto()
    RATE_LIMITED = auto()
    PORTAL_BLOCKED = auto()
    SCHEMA_MISMATCH = auto()
    PERMANENT_FAILURE = auto()


@dataclass(frozen=True)
class ComplianceTask:
    entity_id: str
    jurisdiction_code: str
    filing_id: str            # idempotency key — one in-flight coroutine per filing
    statutory_deadline: datetime
    max_attempts: int = 5


@dataclass
class TokenBucket:
    capacity: float
    tokens: float
    refill_rate: float        # tokens per second, calibrated per jurisdiction
    last_refill: float = field(default_factory=time.monotonic)

    def consume(self, tokens: float = 1.0) -> bool:
        now = time.monotonic()
        self.tokens = min(self.capacity, self.tokens + (now - self.last_refill) * self.refill_rate)
        self.last_refill = now
        if self.tokens >= tokens:
            self.tokens -= tokens
            return True
        return False

Phase 2 — Calibrate a token bucket per jurisdiction

A single limiter holds one bucket per jurisdiction code, with conservative defaults and explicit overrides for portals whose observed ceilings differ. acquire blocks cooperatively until a token is available, so callers never have to reason about pacing themselves.

class JurisdictionRateLimiter:
    def __init__(
        self,
        default_rate: float = 0.5,                          # 1 request / 2s baseline
        jurisdiction_overrides: dict[str, float] | None = None,
    ) -> None:
        self.buckets: dict[str, TokenBucket] = {}
        self.default_rate = default_rate
        self.overrides = jurisdiction_overrides or {}

    def get_bucket(self, jurisdiction: str) -> TokenBucket:
        if jurisdiction not in self.buckets:
            rate = self.overrides.get(jurisdiction, self.default_rate)
            self.buckets[jurisdiction] = TokenBucket(capacity=3.0, tokens=3.0, refill_rate=rate)
        return self.buckets[jurisdiction]

    async def acquire(self, jurisdiction: str) -> None:
        bucket = self.get_bucket(jurisdiction)
        while not bucket.consume():
            await asyncio.sleep(0.25)
            logger.debug(
                "rate_limiter.wait",
                extra={"jurisdiction": jurisdiction, "tokens": round(bucket.tokens, 3)},
            )

Phase 3 — Classify portal responses deterministically

State portals rarely return machine-readable error bodies, so classification reads both status code and body text. This taxonomy is the local view of the shared one documented in Error Categorization & Retry Logic; the PORTAL_BLOCKED branch is what later hands a request off to Headless Browser Fallback Strategies.

def classify_error(status: int, body: str) -> ComplianceErrorCategory:
    text = body.lower()
    match status:
        case 429:
            return ComplianceErrorCategory.RATE_LIMITED
        case 500 | 502 | 503 | 504:
            return ComplianceErrorCategory.TRANSIENT_NETWORK
        case 400 if "schema" in text:
            return ComplianceErrorCategory.SCHEMA_MISMATCH
        case _ if "captcha" in text or "blocked" in text:
            return ComplianceErrorCategory.PORTAL_BLOCKED
        case _:
            return ComplianceErrorCategory.PERMANENT_FAILURE

Phase 4 — Poll with deadline-aware backoff

The poll loop acquires a token, issues the request, and then chooses a wait strategy by remaining compliance window. When a deadline is inside the urgency buffer, a rate limit triggers a short fixed wait on the priority channel instead of unbounded exponential backoff — meeting the statutory clock without hammering the portal. Backoff calibration is covered in depth in Implementing Exponential Backoff for Secretary of State APIs.

import aiohttp


class CompliancePoller:
    URGENCY_BUFFER = timedelta(hours=48)   # within this window, prioritize over backoff

    def __init__(self, rate_limiter: JurisdictionRateLimiter, session: aiohttp.ClientSession) -> None:
        self.rate_limiter = rate_limiter
        self.session = session
        self._active: dict[str, asyncio.Task[dict[str, object]]] = {}

    async def _poll_entity(self, task: ComplianceTask) -> dict[str, object]:
        is_urgent = task.statutory_deadline - datetime.now(timezone.utc) <= self.URGENCY_BUFFER
        url = f"https://api.{task.jurisdiction_code}.sos.gov/entities/{task.entity_id}/status"

        for attempt in range(1, task.max_attempts + 1):
            await self.rate_limiter.acquire(task.jurisdiction_code)
            try:
                async with self.session.get(url, timeout=aiohttp.ClientTimeout(total=15)) as resp:
                    body = await resp.text()
                    category = classify_error(resp.status, body)

                    if resp.status == 200:
                        return {"status": ComplianceStatus.ACTIVE, "payload": body, "category": category}

                    if category is ComplianceErrorCategory.RATE_LIMITED:
                        wait = 5.0 if is_urgent else min(2 ** attempt, 60)
                        logger.warning(
                            "poll.rate_limited",
                            extra={"filing_id": task.filing_id, "attempt": attempt,
                                   "wait_s": wait, "urgent": is_urgent},
                        )
                        await asyncio.sleep(wait)
                        continue

                    if category in (ComplianceErrorCategory.TRANSIENT_NETWORK,
                                    ComplianceErrorCategory.SCHEMA_MISMATCH):
                        await asyncio.sleep(2.0)
                        continue

                    # PORTAL_BLOCKED / PERMANENT_FAILURE — stop spending retry budget
                    return {"status": ComplianceStatus.PENDING_REVIEW, "payload": body, "category": category}

            except asyncio.TimeoutError:
                logger.error("poll.timeout",
                             extra={"filing_id": task.filing_id, "jurisdiction": task.jurisdiction_code})
                await asyncio.sleep(3.0)
            except aiohttp.ClientError as exc:
                logger.exception("poll.client_error",
                                 extra={"filing_id": task.filing_id, "error": str(exc)})
                break

        return {"status": ComplianceStatus.DELINQUENT, "payload": None,
                "category": ComplianceErrorCategory.PERMANENT_FAILURE}

Phase 5 — Enforce single-intent execution

enqueue_and_poll is the only public entry point. It collapses concurrent requests for the same filing_id onto one coroutine, guaranteeing a status transition is captured exactly once even when several workers race on the same renewal.

    async def enqueue_and_poll(self, task: ComplianceTask) -> dict[str, object]:
        if task.filing_id in self._active:
            logger.info("poll.dedup", extra={"filing_id": task.filing_id})
            return await self._active[task.filing_id]

        self._active[task.filing_id] = asyncio.create_task(self._poll_entity(task))
        try:
            return await self._active[task.filing_id]
        finally:
            self._active.pop(task.filing_id, None)

Because ComplianceTask is frozen, attempt counting lives in the loop variable rather than on the message — the immutable task stays a safe, hashable key while mutable pacing state stays in the poller.

Edge Cases and Portal-Specific Gotchas

Rate-limit behavior is not uniform across jurisdictions; calibrate jurisdiction_overrides and the urgency buffer per portal rather than assuming a single profile.

Jurisdiction Portal Observed limiting behavior Recommended refill_rate Gotcha
Delaware (DE) Division of Corporations Session-based caps; degraded data instead of 429 under load 0.33 (1 / 3s) Peak Feb–Mar franchise-tax window; widen urgency buffer to 72h
California (CA) bizfileOnline Hard 429 with no Retry-After; CAPTCHA on burst 0.25 (1 / 4s) CAPTCHA → PORTAL_BLOCKED, route to headless rather than retrying
New York (NY) DOS e-Statement Silent session-token invalidation, returns 200 with empty body 0.5 (1 / 2s) Assert non-empty body; an empty 200 is not “ACTIVE”
Texas (TX) SOSDirect / Comptroller 503 during nightly maintenance windows 0.4 (1 / 2.5s) Treat scheduled-maintenance 503 as transient, not blocked

Verification and Testing

Pacing and idempotency are deterministic, so they can be unit-tested without touching a live portal. Drive the token bucket against a fake monotonic clock and assert dedup collapses concurrent calls.

import pytest


def test_token_bucket_refills_over_time(monkeypatch) -> None:
    clock = iter([0.0, 0.0, 4.0])  # start, init refill, +4s
    monkeypatch.setattr(time, "monotonic", lambda: next(clock))
    bucket = TokenBucket(capacity=3.0, tokens=0.0, refill_rate=0.5)
    assert bucket.consume() is True   # 4s * 0.5 = 2 tokens available


@pytest.mark.asyncio
async def test_single_intent_dedup(monkeypatch) -> None:
    poller = CompliancePoller(JurisdictionRateLimiter(), session=object())  # type: ignore[arg-type]
    calls = 0

    async def fake_poll(task: ComplianceTask) -> dict[str, object]:
        nonlocal calls
        calls += 1
        await asyncio.sleep(0.01)
        return {"status": ComplianceStatus.ACTIVE}

    monkeypatch.setattr(poller, "_poll_entity", fake_poll)
    task = ComplianceTask("e1", "de", "f1", datetime.now(timezone.utc) + timedelta(days=10))
    await asyncio.gather(*(poller.enqueue_and_poll(task) for _ in range(5)))
    assert calls == 1   # five concurrent requests collapsed to one

For integration coverage, point the poller at a mock portal (e.g. aioresponses) that returns a scripted 429 → 429 → 200 sequence and assert the urgent path waits 5s, not exponential, when the deadline is inside the buffer.

Troubleshooting

A bulk sweep gets the whole egress pool blocked partway through
Concurrency is exceeding the portal's real ceiling. Lower that jurisdiction's refill_rate and cap concurrent tasks to the bucket capacity, and add full jitter to the backoff so a recovering portal never sees a synchronized retry burst. Confirm egress is partitioned per jurisdiction so one blocked portal does not take the others down with it.
Deadline-critical entities are timing out behind a large sweep
The urgent path is not being taken. Verify statutory_deadline is populated from State Filing Deadline Calendars and is timezone-aware (UTC). If deadlines arrive naive, the URGENCY_BUFFER comparison silently fails and urgent tasks fall back to exponential backoff.
A New York entity shows ACTIVE but the filing is actually overdue
NY DOS returns 200 with an empty body on session-token invalidation. Treating 200 as success without asserting a non-empty, schema-valid body emits a false ACTIVE. Add a body assertion before resolving and route empty responses to PENDING_REVIEW for manual legal review.
Retries are exhausting budget on a portal that will never recover the request
A PORTAL_BLOCKED or PERMANENT_FAILURE is being misclassified as transient. Check classify_error against the live response body; CAPTCHA and bot-mitigation pages should map to PORTAL_BLOCKED and hand off to Headless Browser Fallback Strategies rather than consuming the retry loop.
The same status transition is recorded twice in the compliance ledger
Two workers polled the same filing without going through enqueue_and_poll, or used different filing_id keys for the same obligation. Single-intent dedup only works when the idempotency key is stable per (jurisdiction, entity, period); regenerate it deterministically rather than per request.

Operational Checklist

Frequently Asked Questions

Why a per-jurisdiction token bucket instead of one global rate limiter?
A global limit is simultaneously too slow for permissive portals and too fast for fragile ones, and it lets one jurisdiction's throttling starve every other. Per-jurisdiction buckets let you calibrate each portal to its own observed ceiling and isolate a blocked portal to its own bucket, which is essential when a bulk sweep spans all fifty states at once.
How does deadline-aware backoff avoid getting the IP blocked?
The urgent path uses a short fixed wait on a low-concurrency priority channel, not zero wait — it still honors the token bucket, it just skips unbounded exponential growth so a deadline-critical check is not deferred for minutes. Only entities inside the urgency buffer take that path, so the volume of prioritized requests stays small enough not to trip portal defenses.
What guarantees a status transition is recorded exactly once?
Single-intent execution: enqueue_and_poll keys in-flight work by filing_id and returns the existing coroutine for any duplicate request. Combined with a deterministic idempotency key per (jurisdiction, entity, period), concurrent workers collapse onto one poll and one ledger write.
When should a rate-limited request stop retrying and escalate?
When the response classifies as PORTAL_BLOCKED or PERMANENT_FAILURE, or when max_attempts is reached. At that point the task resolves to PENDING_REVIEW or DELINQUENT and emits a compliance alert rather than burning compute — blocked portals are routed to headless fallback, and unresolved deadlines escalate to legal operations.