Hiding Two Incompatible Pagination Styles Behind One Correct Interface #

This guide belongs to the Pagination Handling for Bulk Records area of the Secretary of State Portal & API Ingestion framework, and it addresses the fact that no two states paginate the same way. Some registries hand you an opaque cursor that pins a stable position in the result set; others expose a raw offset/limit grid that indexes by row number. These are not two flavours of the same thing — they have different correctness guarantees under concurrent writes — and a bulk-ingestion pipeline that hard-codes one style per portal accumulates a maze of special cases. This page builds a single abstraction over both, and is candid about why the offset style is the dangerous one.

Scope of This Page #

This page covers the design of one pagination interface that both an opaque-cursor portal and an offset/limit portal can satisfy, and the correctness reasoning that makes that abstraction honest rather than a leaky lowest-common-denominator. It walks through why offset pagination silently skips or duplicates rows when the underlying result set changes mid-walk — the failure mode that makes it structurally weaker than a cursor — and how a strategy-agnostic consumer defends itself regardless of which style it is handed. It excludes the portal-specific quirks of any single registry; the deep treatment of California’s page-number grid lives in Paginating Bulk Entity Searches on California BizFile, and the transport concerns of pacing and retrying page requests belong to Async Polling & Rate Limiting and Error Categorization & Retry Logic. Here we assume a paced, retried request and focus purely on the shape of the paging contract.

The Correctness Difference That Justifies the Abstraction #

The reason to build one interface is not tidiness — it is that the two styles fail differently, and the consumer must be written to survive the weaker one. An opaque cursor encodes a stable key (a high-water entity id, a keyset position) so that “the next page” means “everything after this key,” which is well-defined even as rows are inserted and deleted around it. An offset/limit request means “rows 200 through 299 of the current ordering,” and the current ordering is re-evaluated on every request. When a new entity is registered — a routine event on any active registry — every row after it shifts down by one position. The row that was at offset 300 is now at offset 301, so a walk that already read offset 0–299 and asks for 300–399 next will re-read one row (a duplicate) and, symmetrically, a deletion pushes a row up past the boundary you already crossed (a skip). Because these registries mutate continuously — Delaware processes franchise-tax filings under 8 Del. C. § 502 year-round, New York accepts biennial statements under N.Y. BCL § 408 daily — offset pagination over a live result set is not merely inelegant; it is a source of the silent under- and over-counting the parent area exists to prevent. The abstraction therefore exposes a cursor-shaped contract and forces the offset implementation to compensate for what it cannot natively guarantee.

Prerequisites #

• Python 3.10+ — for Protocol, X | None unions, and async iteration.
• httpx 0.27+ — one pooled async client shared by both paginator implementations.
• A stable business key per record (the entity id) to deduplicate on — the offset path depends on it.
• An understanding that pagination style is a portal property, discovered once and encoded in the adapter, not re-decided per request.
• Paced, retried requests: pacing and fault classification are handled by the layers named above, not here.

Implementation: A Paginator Protocol With Two Strategies #

The module defines a Paginator protocol whose whole surface is fetch(token) -> Page. A CursorPaginator returns the portal’s opaque next_cursor as the token; an OffsetPaginator returns the next absolute offset as the token. The consumer — walk — never learns which it is holding, and deduplicates on the entity id so that the offset strategy’s inevitable re-served row is dropped rather than emitted. The diagram shows both strategies collapsing into the one interface the consumer talks to.

from __future__ import annotations

import json
import logging
from dataclasses import dataclass
from typing import Any, AsyncIterator, Protocol, runtime_checkable

import httpx

# Structured JSON logging — every line is a parseable audit/observability event.
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("ingestion.paginator")


@dataclass(frozen=True)
class Page:
    rows: list[dict[str, Any]]
    has_next: bool
    # The token that fetches the NEXT page. Opaque for cursor portals, an absolute
    # offset for offset portals. The consumer treats it as a black box either way.
    next_token: str | None


@runtime_checkable
class Paginator(Protocol):
    """One page-walking contract. Implementations hide whether the portal is cursor- or offset-paged."""

    jurisdiction: str

    async def fetch(self, token: str | None) -> Page: ...


class CursorPaginator:
    """Opaque-cursor portals: a modern JSON API that returns next_cursor with each page."""

    def __init__(self, client: httpx.AsyncClient, path: str, jurisdiction: str) -> None:
        self._client = client
        self._path = path
        self.jurisdiction = jurisdiction

    async def fetch(self, token: str | None) -> Page:
        params = {"cursor": token} if token else {}
        resp = await self._client.get(self._path, params=params)
        resp.raise_for_status()
        body = resp.json()
        cursor = body.get("next_cursor")
        rows = body.get("entities", [])
        logger.info(json.dumps({"event": "cursor_page", "jurisdiction": self.jurisdiction,
                                "rows": len(rows), "has_next": cursor is not None}))
        # A cursor pins a stable key: a concurrent write cannot shift what the cursor points past.
        return Page(rows=rows, has_next=cursor is not None, next_token=cursor)


class OffsetPaginator:
    """Offset/limit portals: legacy result grids. Position-based, so unsafe under concurrent writes."""

    def __init__(self, client: httpx.AsyncClient, path: str, jurisdiction: str, limit: int = 100) -> None:
        self._client = client
        self._path = path
        self.jurisdiction = jurisdiction
        self._limit = limit

    async def fetch(self, token: str | None) -> Page:
        offset = int(token) if token else 0
        params = {
            "offset": offset,
            "limit": self._limit,
            "sort": "entity_id",   # a fixed sort bounds — but does NOT eliminate — the shift hazard
        }
        resp = await self._client.get(self._path, params=params)
        resp.raise_for_status()
        rows = resp.json().get("entities", [])
        # A full page implies there may be more; a short page is the natural end.
        has_next = len(rows) == self._limit
        next_offset = str(offset + self._limit) if has_next else None
        logger.info(json.dumps({"event": "offset_page", "jurisdiction": self.jurisdiction,
                                "offset": offset, "rows": len(rows), "has_next": has_next}))
        return Page(rows=rows, has_next=has_next, next_token=next_offset)


async def walk(paginator: Paginator, dedupe_key: str = "entity_id") -> AsyncIterator[dict[str, Any]]:
    """Strategy-agnostic driver — identical code path for cursor and offset portals.

    Deduping on a stable business key is what lets the SAME loop stay correct over the
    offset strategy, which will re-serve a row whenever the result set shifts mid-walk.
    """
    seen: set[str] = set()
    token: str | None = None
    page_count = 0
    while True:
        page = await paginator.fetch(token)
        for row in page.rows:
            key = row[dedupe_key]                     # identity, never row position
            if key in seen:
                # The offset path lands here after a concurrent insert shifts the window.
                logger.warning(json.dumps({"event": "duplicate_skipped",
                                           "jurisdiction": paginator.jurisdiction, "key": key}))
                continue
            seen.add(key)
            yield row
        page_count += 1
        if not page.has_next:
            logger.info(json.dumps({"event": "walk_complete", "jurisdiction": paginator.jurisdiction,
                                    "pages": page_count, "unique_rows": len(seen)}))
            return
        token = page.next_token

The Protocol is the whole point: walk depends only on fetch(token) -> Page, so a portal that upgrades from an offset grid to a proper cursor API is a one-line swap of the implementation, with no change to the consumer or the tests around it. What the abstraction cannot do is upgrade the guarantee — an OffsetPaginator is still position-based underneath, so the dedupe_key check exists precisely to absorb the duplicate it will eventually produce. A rising duplicate_skipped rate on an offset portal is the operational tell that the registry is churning fast enough that you should be pressing the state for a cursor or keyset endpoint.

Jurisdiction Pagination Map #

The style is a fixed property of each portal, discovered once and encoded in the adapter you construct for it.

Three of the four major registries page by position, which is why the offset strategy is the common case rather than the exception, and why the consumer is written to defend against it by default rather than trusting any portal to hand back a clean, stable sequence.

Failure Modes and Fallback Routing #

Each maps onto the four-tier scheme in the parent area’s error categorization & retry logic — transient, statutory, data-validation, and system.

1. Concurrent insert duplicates a row on an offset portal (system, absorbed). A new registration shifts every later offset, so the next page re-serves the boundary row. The seen set drops it and logs duplicate_skipped; the walk stays correct. Sustained duplication is the signal to migrate that portal to a keyset or cursor endpoint if one exists.
2. Concurrent delete skips a row on an offset portal (system, the dangerous case). A deletion pulls a row up past a boundary you have already crossed, so it is never served — a silent gap that dedupe cannot detect because there is nothing to compare against. This is why offset walks must not stand alone as a completeness assertion; the parent area’s completeness gate, comparing the unique count to a jurisdictional baseline, is what catches the shortfall a cursor would have prevented outright.
3. An offset portal caps its depth (system, needs narrowing). Legacy grids (Delaware, Texas) stop serving new rows past a fixed offset and may repeat the last page. has_next keeps returning true while the rows stop being new, so dedupe suppresses the repeats and the walk terminates on the eventual short or all-duplicate page — but a query that hits the cap must be sharded, exactly as documented for California BizFile.
4. A row is missing the dedupe key (data-validation). row[dedupe_key] raises rather than admitting a record with no stable identity into the walk, since such a record can be neither deduplicated nor counted. Quarantine it against your compliance metadata schemas and continue the sweep.

Frequently Asked Questions #

Related #

• Pagination Handling for Bulk Records
• Paginating Bulk Entity Searches on California BizFile
• Error Categorization & Retry Logic
• Async Polling & Rate Limiting
• Secretary of State Portal & API Ingestion