Retrieving Every Entity From a State Registry Without Silently Dropping Pages #

This guide is part of the Secretary of State Portal & API Ingestion discipline, and it solves one narrow but consequential problem: when you sweep a state registry for hundreds or thousands of entities, you must be able to prove you retrieved all of them. A pagination loop that stops one page early does not raise an exception — it returns a clean, plausible, incomplete dataset. Downstream, that gap becomes an entity that never gets a deadline, a missed annual report, and an administrative dissolution. Pagination handling is therefore not a transport convenience; it is a completeness control with the same evidentiary weight as the Compliance Metadata Schemas it feeds.

Statutory and Regulatory Context #

Bulk record retrieval underpins the most basic compliance assertion a legal operations team makes: this is the complete set of entities we are responsible for in this jurisdiction. Most states publish entity data subject to public-records statutes — California’s bulk data offering derives from the California Public Records Act and the BizFile portal operated under Cal. Corp. Code, while Delaware’s Division of Corporations exposes entity status under the disclosure regime of Del. Code tit. 8. None of these portals contractually guarantees stable pagination; cursors expire, total-count headers drift mid-sweep as the registry mutates, and legacy result grids cap at a fixed page depth regardless of how many entities actually match.

The compliance consequence is asymmetric. An over-inclusive sweep is merely wasteful — you poll a few entities you do not own. An under-inclusive sweep is a liability event: the entity you failed to page in is the entity whose franchise tax goes unpaid. Because corporate counsel may later be asked to demonstrate the registry snapshot on a specific date as evidence of due diligence, every traversal must record not only the records it collected but the reason it believed the collection was complete. That turns pagination from a while loop into an auditable state machine.

Architecture and Design Model #

The design treats one full traversal of one jurisdiction as a single atomic transaction with an explicit, recorded termination reason — never an open-ended sequence of independent HTTP calls. Five decisions follow from that:

• Cursor state is immutable context, not a mutable counter. Each request carries the jurisdiction code, filing period, and entity-type filter so a replayed or resumed sweep is deterministic.
• Termination is enumerated, not inferred. The loop ends only by hitting a named PaginationTerminationReason (complete, empty set, expired token, circuit open, completeness gate failed). “The loop ran out of iterations” is itself a recorded — and alertable — outcome.
• Completeness is gated against a baseline. Retrieved volume is compared to a historical per-jurisdiction expectation before the data is allowed downstream, so a silently truncated sweep is caught at the source rather than weeks later.
• Fallback is layered, not branching. When the JSON cursor contract breaks, the same traversal intent is satisfied by Headless Browser Fallback Strategies, then by a human review queue — each layer preserving the original page sequence.
• Memory is bounded by streaming. Records are yielded in chunks, never accumulated, so a 40,000-entity Texas sweep has the same footprint as a 200-entity one.

This places pagination squarely between the portal adapters above it and the multi-entity batch orchestration layer that consumes complete entity sets below it.

Prerequisites and Dependencies #

Infrastructure assumptions: egress identity is partitioned per jurisdiction (a block on one sweep must not cascade), and circuit-breaker state is persisted in a distributed cache so an in-progress traversal survives a worker restart idempotently.

Step-by-Step Implementation #

Phase 1 — Model immutable context and recorded termination #

Pagination state separates the intent (frozen context) from the progress (mutable cursor and audit log). The termination reason is a first-class enum so every exit is explainable.

from __future__ import annotations

import logging
from dataclasses import dataclass, field
from datetime import datetime, timezone
from enum import Enum, auto
from typing import Any

logger = logging.getLogger("pagination")


class PaginationTerminationReason(Enum):
    COMPLETE = auto()
    EMPTY_RESULT_SET = auto()
    JURISDICTION_TOKEN_EXPIRED = auto()
    CIRCUIT_BREAKER_OPEN = auto()
    COMPLETENESS_GATE_FAILED = auto()
    PAGE_LIMIT_REACHED = auto()  # an explicit, alertable outcome — never silent


@dataclass(frozen=True)
class PaginationContext:
    jurisdiction: str
    filing_period: str
    entity_type_filter: str
    page_size: int = 100
    max_pages: int = 500


@dataclass
class TraversalState:
    context: PaginationContext
    cursor: str | None = None
    page_index: int = 0
    total_retrieved: int = 0
    termination_reason: PaginationTerminationReason | None = None
    audit_log: list[dict[str, Any]] = field(default_factory=list)

    def record_page(self, count: int, cursor: str | None) -> None:
        self.page_index += 1
        self.total_retrieved += count
        self.cursor = cursor
        self.audit_log.append(
            {
                "event": "page_fetched",
                "jurisdiction": self.context.jurisdiction,
                "page": self.page_index,
                "records_fetched": count,
                "cursor_present": cursor is not None,
                "timestamp": datetime.now(timezone.utc).isoformat(),
            }
        )

Phase 2 — Drive the cursor loop with deterministic exits #

The loop never falls off the end ambiguously. It terminates on an empty page, a missing next cursor, a transport failure, or the page limit — and each branch sets a reason.

import httpx


class CompliancePaginationEngine:
    """Deterministic cursor traversal for one jurisdiction, one filing period."""

    def __init__(self, base_url: str, timeout: float = 30.0) -> None:
        self._base_url = base_url.rstrip("/")
        # Connection pooling + keep-alive: reuse TLS across hundreds of pages.
        self._client = httpx.Client(
            base_url=self._base_url,
            timeout=timeout,
            limits=httpx.Limits(max_keepalive_connections=10),
        )

    def traverse(self, context: PaginationContext) -> TraversalState:
        state = TraversalState(context=context)
        logger.info(
            "traversal_start",
            extra={"jurisdiction": context.jurisdiction, "period": context.filing_period},
        )
        while state.page_index < context.max_pages:
            try:
                payload = self._fetch_page(state)
            except httpx.HTTPError as exc:
                logger.error("page_request_failed", extra={"error": str(exc)})
                state.termination_reason = PaginationTerminationReason.CIRCUIT_BREAKER_OPEN
                return state

            records = payload.get("entities", [])
            if not records:
                state.termination_reason = PaginationTerminationReason.EMPTY_RESULT_SET
                break

            next_cursor = payload.get("next_cursor")
            state.record_page(len(records), next_cursor)
            yield_chunk(records)  # hand off immediately; never accumulate

            if next_cursor is None:
                state.termination_reason = PaginationTerminationReason.COMPLETE
                break
        else:
            # Loop exhausted max_pages without a natural end — treat as a fault.
            state.termination_reason = PaginationTerminationReason.PAGE_LIMIT_REACHED
            logger.warning("page_limit_reached", extra={"pages": state.page_index})

        return state

    def _fetch_page(self, state: TraversalState) -> dict[str, Any]:
        params = {
            "jurisdiction": state.context.jurisdiction,
            "filing_period": state.context.filing_period,
            "entity_type": state.context.entity_type_filter,
            "limit": state.context.page_size,
            "cursor": state.cursor,  # None on the first request
        }
        # Timeout is per-request on the client, not a Session attribute.
        response = self._client.get("/entities", params=params)
        response.raise_for_status()
        return response.json()

A note that bites teams migrating from requests: there is no session.timeout attribute on a requests.Session, and assigning one is silently ignored. Timeouts must be passed per-request (httpx.Client(timeout=...) here, or session.get(..., timeout=...) in requests). A “hung” sweep is almost always a timeout that was set on the wrong object.

Phase 3 — Gate completeness against a jurisdictional baseline #

Before any page set is allowed downstream, compare the retrieved volume to a historical expectation. A delta beyond the threshold is treated as a truncated sweep, not a real shrinkage of the registry.

class CompletenessGate:
    def __init__(self, baselines: dict[str, int], tolerance: float = 0.15) -> None:
        self._baselines = baselines
        self._tolerance = tolerance  # 15% — tune per jurisdiction volatility

    def evaluate(self, state: TraversalState) -> bool:
        baseline = self._baselines.get(state.context.jurisdiction)
        if baseline is None:
            logger.info("no_baseline_skip_gate", extra={"jurisdiction": state.context.jurisdiction})
            return True  # first sweep establishes the baseline; do not block
        delta = abs(state.total_retrieved - baseline) / max(baseline, 1)
        if delta > self._tolerance:
            state.termination_reason = PaginationTerminationReason.COMPLETENESS_GATE_FAILED
            logger.warning(
                "completeness_gate_failed",
                extra={
                    "jurisdiction": state.context.jurisdiction,
                    "retrieved": state.total_retrieved,
                    "baseline": baseline,
                    "delta_pct": round(delta * 100, 2),
                },
            )
            return False
        return True

The gate is deliberately fail-closed: a failed sweep raises a pre-filing audit hold rather than emitting partial data into State Filing Deadline Calendars. A genuine registry change (a mass dissolution, a new filing wave) is reconciled by an operator who then updates the baseline — the gate surfaces the event instead of swallowing it.

Phase 4 — Stream records under a constant memory ceiling #

yield_chunk is where bounded memory is enforced. Records flow to the consumer in fixed batches and are never held in a growing list, so portfolio size does not drive the worker’s heap.

from collections.abc import Iterator
from itertools import islice


def chunked(records: list[dict[str, Any]], size: int = 500) -> Iterator[list[dict[str, Any]]]:
    it = iter(records)
    while batch := list(islice(it, size)):
        yield batch


def yield_chunk(records: list[dict[str, Any]]) -> None:
    for batch in chunked(records, size=500):
        downstream_filing_processor.submit(batch)  # back-pressured queue

Edge Cases and Portal-Specific Gotchas #

Pagination contracts vary more than any other part of ingestion. The completeness gate is what makes these survivable, but each portal needs a calibrated adapter. Where a portal departs from cursor-based JSON, traversal degrades to Headless Browser Fallback Strategies.

For the JS-rendered and irregular-table cases, the deterministic extraction patterns live in Parsing inconsistent HTML tables from legacy state portals, which maps irregular layouts onto the same record contract this engine validates.

Verification and Testing #

Assert two properties: that the loop terminates for the right reason, and that the completeness gate fails closed.

import pytest


class _StubClient:
    """Serves a fixed sequence of pages, then a cursorless final page."""

    def __init__(self, pages: list[dict[str, Any]]) -> None:
        self._pages = pages
        self._i = 0

    def get(self, _path: str, params: dict[str, Any]):  # noqa: ANN201
        page = self._pages[self._i]
        self._i += 1
        return _StubResponse(page)


def test_traversal_terminates_complete_on_null_cursor() -> None:
    engine = CompliancePaginationEngine("https://example.test")
    engine._client = _StubClient(
        [
            {"entities": [{"id": "1"}, {"id": "2"}], "next_cursor": "c2"},
            {"entities": [{"id": "3"}], "next_cursor": None},
        ]
    )
    ctx = PaginationContext(jurisdiction="DE", filing_period="2026", entity_type_filter="LLC")
    state = engine.traverse(ctx)
    assert state.termination_reason is PaginationTerminationReason.COMPLETE
    assert state.total_retrieved == 3
    assert len(state.audit_log) == 2  # one record_page per fetched page


def test_completeness_gate_fails_closed_on_truncation() -> None:
    gate = CompletenessGate(baselines={"CA": 1200}, tolerance=0.15)
    state = TraversalState(
        context=PaginationContext("CA", "2026", "CORP"), total_retrieved=400
    )
    assert gate.evaluate(state) is False
    assert state.termination_reason is PaginationTerminationReason.COMPLETENESS_GATE_FAILED

For integration coverage, point the engine at a recorded fixture of real portal pages (VCR-style cassettes) so cursor-expiry and repeated-last-page behavior are exercised without live traffic. Seed the baseline store from the prior period’s verified counts.

Troubleshooting #

Operational Checklist #

• Every traversal exits with an explicit PaginationTerminationReason — PAGE_LIMIT_REACHED is alerted, never ignored
• Per-request timeouts are set on the client, not on a Session attribute
• Transport retries are bounded (429/5xx) so failures trip the circuit breaker, not an infinite loop
• A per-jurisdiction baseline exists and the completeness gate fails closed when the delta exceeds tolerance
• Each page payload is validated against the record contract before its count is trusted
• Records stream through fixed-size chunks; no full result set is held in memory
• Offset-grid jurisdictions (DE, TX) dedupe on entity id to catch repeated last pages
• Session-bound portals (NY) persist cookies and resume from the last recorded page_index
• JS-rendered / irregular-table portals route to headless extraction with token reconstruction
• The full audit_log — page index, counts, UTC timestamps, termination reason — is written to the compliance ledger before any filing prep begins

Related #

• Secretary of State Portal & API Ingestion
• Parsing Inconsistent HTML Tables From Legacy State Portals
• Headless Browser Fallback Strategies
• Async Polling & Rate Limiting
• Error Categorization & Retry Logic