Secretary Of State Portal Api Ingestion

Maintaining Browser Sessions Across State Portal Timeouts

This guide is part of the Headless Browser Fallback Strategies area within the Secretary of State Portal & API Ingestion framework. Where the rest of that area keeps each browser read short and single-intent, this page handles the case a long authenticated sweep cannot avoid: a legacy state portal that silently invalidates the server-side session out from under you, mid-read, and how to persist, probe, and deterministically re-establish that session so the sweep resumes instead of recording a false logged-out state.

Scope of This Page

This page covers the lifecycle of one authenticated headless session against a portal with an aggressive server-side idle timeout: persisting the authenticated cookies and local storage with Playwright’s storage_state, restoring them on restart to skip a full login, firing keep-alive heartbeats before the idle window closes, detecting a silent logout — the server dropped the session but the page still looks logged in until the next navigation — and re-authenticating deterministically from a known state. It excludes the initial login form mechanics for each portal and the read logic layered on top; the concern here is purely session survival. Anti-idle heartbeats are a keep-alive control, not an evasion — a portal that hard-blocks automation is still a terminal boundary, handled by the parent area.

The Operational Constraint: Timeouts Outlive Reads

The tension is structural. A multi-entity compliance sweep against an authenticated portal — pulling filing history or good-standing detail for hundreds of entities — takes far longer than the portal’s idle window allows for a single session, and re-logging in for every entity is both slow and a reliable way to trip a rate limit or a challenge. Yet the underlying obligation is unforgiving: a session that dies mid-sweep and is misread as “entity not found” can hide a delinquent filing, and jurisdictions grant no slack for that. New York’s biennial Statement under N.Y. BCL § 408 carries no grace window; California moves a corporation toward suspension of powers under Cal. Corp. Code § 2205 when its Statement of Information under § 1502 lapses; Delaware’s franchise-tax and annual-report obligation under 8 Del. C. § 502 accrues penalty and interest under § 510. Because a wrong “logged out” answer can mask any of these, the session layer must distinguish a genuine empty result from a dead session with certainty — which means an explicit health probe, not an inference from a blank page.

Portal Timeout Behavior by Jurisdiction

The four jurisdictions differ enough that a single keep-alive cadence fails against at least one of them. New York is the tightest and drives the default heartbeat interval; Texas is the most forgiving but sits behind an authenticated, paywalled account.

Jurisdiction Portal Idle timeout (typical) Silent-logout signal Filing anchor
Delaware Division of Corporations (iCIS) ~20 min Next POST returns the login form in-page 8 Del. C. § 502 / § 510
California BizFile Online ~15 min SPA XHR returns 401; app redirects to login Cal. Corp. Code § 1502 / § 2205
New York DOS Public Inquiry ~5 min Nav redirects to a login link; grid never renders N.Y. BCL § 408
Texas SOSDirect ~30 min Session token cleared; account-login page served Tex. BOC § 4.002 / Tax Code § 171

Prerequisites

  • Python 3.10+ — for X | Y unions and modern asyncio.
  • Playwright (Python) 1.40+ — storage_state save/restore and explicit waits; playwright install chromium once.
  • Standard library beyond that: asyncio, json, logging, time, os, pathlib, dataclasses, enum.
  • A secret store supplying portal credentials via environment (never hard-coded) and a durable, access-controlled path for the persisted storage_state file, which holds live session cookies and must be treated as a secret.
  • One authenticated portal profile per jurisdiction — login URL, an authenticated-home URL, and selectors that distinguish a logged-in page from a logged-out one.

Implementation: A Session Lifecycle Keeper

The session moves through four states, and the whole design is about making every transition deterministic and observed rather than assumed. An ACTIVE session drifts toward IDLE-WARN as the idle clock runs down; a heartbeat that lands in time returns it to ACTIVE, but if the server has already dropped it the health probe sees the logout marker and the session enters EXPIRED; from there a deterministic re-login (RE-AUTH) restores authenticated state and the sweep resumes. The diagram shows that cycle; the module beneath it implements it with storage_state persistence, a health probe that navigates to an authenticated page and checks for a logout marker, and a heartbeat gated on the idle margin.

Session lifecycle across a portal idle timeout: ACTIVE, IDLE-WARN, EXPIRED, RE-AUTH, and back to ACTIVE A four-state machine. ACTIVE drifts to IDLE-WARN past the warn margin; a heartbeat loops IDLE-WARN back to ACTIVE, but a detected silent logout sends it to EXPIRED, then RE-AUTH performs a deterministic re-login and restores persisted storage_state, looping back to ACTIVE when the health probe passes. One session across a portal idle timeout: heartbeat to stay alive, re-auth deterministically when dropped ACTIVE session healthy storage_state persisted IDLE-WARN near idle timeout heartbeat fires EXPIRED silent server logout probe sees login marker RE-AUTH deterministic re-login restore cookies + storage idle > warn silent logout re-auth heartbeat lands in time → keep alive health probe passes → storage_state restored, resume sweep storage_state file underlies the cycle · restart resumes without a fresh login 
from __future__ import annotations

import asyncio
import json
import logging
import os
import time
from dataclasses import dataclass
from enum import Enum
from pathlib import Path

from playwright.async_api import Browser, BrowserContext, async_playwright

# Structured JSON logging — every transition is an auditable observability event.
logging.basicConfig(level=logging.INFO, format="%(message)s")
logger = logging.getLogger("ingestion.session_keeper")


class SessionState(str, Enum):
    ACTIVE = "active"
    IDLE_WARN = "idle_warn"
    EXPIRED = "expired"
    REAUTH = "reauth"


@dataclass(frozen=True)
class PortalProfile:
    jurisdiction: str            # ISO 3166-2, e.g. "US-NY"
    login_url: str
    home_url: str                # an authenticated landing page
    idle_timeout_s: float        # server-side idle window before a silent logout
    warn_margin_s: float         # heartbeat this many seconds before the timeout
    logged_in_selector: str      # present only while authenticated
    logged_out_selector: str     # a login form / redirect marker => session died


# Idle windows differ sharply by jurisdiction; NY (300 s) drives the tightest cadence.
PROFILES: dict[str, PortalProfile] = {
    "US-DE": PortalProfile("US-DE", "https://icis.corp.delaware.gov/login",
                           "https://icis.corp.delaware.gov/ecorp/", 1200, 120,
                           "#sessionUser", "form#loginForm"),
    "US-CA": PortalProfile("US-CA", "https://bizfileonline.sos.ca.gov/login",
                           "https://bizfileonline.sos.ca.gov/search", 900, 90,
                           "[data-account-menu]", "input[name='password']"),
    "US-NY": PortalProfile("US-NY", "https://apps.dos.ny.gov/login",
                           "https://apps.dos.ny.gov/publicInquiry/", 300, 45,
                           "#authedNav", "a[href*='login']"),
    "US-TX": PortalProfile("US-TX", "https://direct.sos.state.tx.us/acct/acct-login.asp",
                           "https://direct.sos.state.tx.us/corp_inquiry/", 1800, 180,
                           "#SignOff", "input[name='client_id']"),
}


class PortalSession:
    """Keeps one authenticated headless session alive across a portal's idle timeout."""

    def __init__(self, profile: PortalProfile, state_path: Path) -> None:
        self._p = profile
        self._state_path = state_path              # persisted cookies + storage; treat as secret
        self._context: BrowserContext | None = None
        self._last_activity = 0.0
        self.state = SessionState.EXPIRED

    def _log(self, event: str, **fields: object) -> None:
        logger.info(json.dumps({"event": event, "jurisdiction": self._p.jurisdiction,
                                "state": self.state.value, **fields}))

    async def start(self, browser: Browser) -> BrowserContext:
        """Restore a persisted session if it is still valid, else log in fresh."""
        if self._state_path.exists():
            ctx = await browser.new_context(storage_state=str(self._state_path))
            if await self._is_authenticated(ctx):
                self._context, self._last_activity = ctx, time.monotonic()
                self.state = SessionState.ACTIVE
                self._log("session_restored")
                return ctx
            await ctx.close()                      # stored state was stale — re-auth cleanly
        self._context = await self._login(browser)
        self._last_activity = time.monotonic()
        self.state = SessionState.ACTIVE
        return self._context

    async def _login(self, browser: Browser) -> BrowserContext:
        """Deterministic re-login; credentials come from the secret store, never code."""
        self.state = SessionState.REAUTH
        ctx = await browser.new_context()
        page = await ctx.new_page()
        await page.goto(self._p.login_url, wait_until="domcontentloaded", timeout=20_000)
        await page.fill("input[name='username']", os.environ["PORTAL_USER"])
        await page.fill("input[name='password']", os.environ["PORTAL_PASSWORD"])
        await page.click("button[type='submit']")
        await page.wait_for_selector(self._p.logged_in_selector, timeout=15_000)
        # Persist authenticated cookies + storage so a process restart skips re-login.
        await ctx.storage_state(path=str(self._state_path))
        await page.close()
        self._log("session_established")
        return ctx

    async def _is_authenticated(self, ctx: BrowserContext) -> bool:
        """Health probe: a logout marker on an authenticated URL means a silent logout."""
        page = await ctx.new_page()
        try:
            await page.goto(self._p.home_url, wait_until="domcontentloaded", timeout=15_000)
            if await page.query_selector(self._p.logged_out_selector) is not None:
                return False                       # server killed the session behind our back
            return await page.query_selector(self._p.logged_in_selector) is not None
        finally:
            await page.close()

    async def heartbeat(self) -> None:
        """Fire before the idle window closes to reset the server's inactivity timer."""
        assert self._context is not None, "call start() before heartbeat()"
        idle = time.monotonic() - self._last_activity
        if idle < self._p.idle_timeout_s - self._p.warn_margin_s:
            return                                 # still comfortably ACTIVE — no-op
        self.state = SessionState.IDLE_WARN
        if await self._is_authenticated(self._context):
            self._last_activity = time.monotonic()
            self.state = SessionState.ACTIVE
            self._log("heartbeat_ok", idle_s=round(idle, 1))
        else:
            self.state = SessionState.EXPIRED
            self._log("silent_logout_detected", idle_s=round(idle, 1))

    async def ensure_alive(self, browser: Browser) -> BrowserContext:
        """Guarantee a live context before a read: probe, and re-auth if the session died."""
        if self._context is None:
            return await self.start(browser)
        if self.state is SessionState.EXPIRED or not await self._is_authenticated(self._context):
            self.state = SessionState.EXPIRED
            self._log("reauth_begin")
            await self._context.close()
            self._context = await self._login(browser)   # full, deterministic recovery
            self._last_activity = time.monotonic()
            self.state = SessionState.ACTIVE
        return self._context


async def main() -> None:
    profile = PROFILES["US-NY"]                    # tightest window of the four (300 s)
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(
            headless=True, args=["--no-sandbox", "--disable-dev-shm-usage"]
        )
        session = PortalSession(profile, Path("/var/lib/ingestion/ny_state.json"))
        await session.start(browser)
        for _ in range(20):                        # a long authenticated sweep
            ctx = await session.ensure_alive(browser)
            page = await ctx.new_page()
            await page.goto(profile.home_url, wait_until="domcontentloaded")
            await page.close()
            await session.heartbeat()              # keep the server timer from firing
            await asyncio.sleep(30)


if __name__ == "__main__":
    asyncio.run(main())

The keeper is deliberately conservative: it never assumes a session is alive because it was alive a minute ago. ensure_alive probes before every read, and heartbeat only touches the portal once the idle margin is reached, so it neither hammers the server nor lets the timer expire unobserved. A silent logout is caught by the same _is_authenticated probe whether it surfaces during a heartbeat or on the next read, which keeps EXPIRED a single, well-defined condition rather than a scatter of edge cases.

Configuration Reference

These values are dictated by each portal’s server-side behavior, not tuned by preference. The margin is the safety buffer that decides whether a heartbeat lands before or after the server gives up on the session.

Parameter Suggested value Operational justification
idle_timeout_s per portal (see table) Must reflect the server’s real idle window; guessing high guarantees silent logouts mid-sweep.
warn_margin_s 15–20% of the timeout Buffer for probe latency and clock jitter so a heartbeat fires with room to spare.
storage_state path durable, access-controlled Holds live session cookies — a resumable session and a secret; loss forces a full re-login.
logged_out_selector login form / redirect marker The single source of truth for a dead session; a blank page alone is not proof of logout.
Heartbeat action authenticated home GET Cheapest round-trip that resets the timer and confirms auth in one navigation.
Credentials environment / secret store Never hard-coded; rotated independently of the persisted storage_state.
ensure_alive cadence before every read Trades one cheap probe for certainty that a not found is real, not a dead session.

Failure Modes and Fallback Routing

Each fault maps onto the four-tier scheme in the parent area’s Error Categorization & Retry Logic — transient, statutory, data-validation, and system — so a session fault is triaged with the same codes as any other ingestion failure.

  1. Silent logout mid-sweep (system, recoverable). The server dropped the session but the last-rendered page still looked authenticated. The _is_authenticated probe on the next ensure_alive finds the logout marker, the session enters EXPIRED, and _login restores it deterministically. The read that triggered recovery is retried on the fresh context, so no entity is skipped.
  2. Stale storage_state on restart (system). A persisted session file outlived the server session. start restores it, the probe fails immediately, the stale context is closed, and a clean login replaces it — the restart costs one login, not a corrupted sweep against a half-dead context.
  3. Repeated re-auth in a tight loop (hard block, escalate). If _login itself keeps landing on a challenge or the credentials are rejected, the portal is refusing automation, not merely timing out. That is not a session problem to retry — it is a hard block the Headless Browser Fallback Strategies area routes to human review, and the keeper must surface the repeated failure rather than spin.
  4. Heartbeats themselves tripping a rate limit (transient). Too tight a heartbeat cadence across many concurrent sessions can look like abuse and earn a throttle. The fix is not more retries in the keeper but pacing the fleet through Async Polling & Rate Limiting, so heartbeats stay under each portal’s tolerance while still landing before the idle window closes.

Frequently Asked Questions

Why probe with a real navigation instead of just checking whether a cookie is still present?

Because a present cookie proves nothing about the server’s view of the session. Legacy portals invalidate the server-side session on their own idle timer while the client still holds a cookie that looks valid, so a cookie check would report ACTIVE for a session the server has already discarded — the exact silent-logout case that misreads an entity as not found. Navigating to an authenticated URL and checking for the logged_out_selector is the only signal that reflects the server’s actual state.

Is a keep-alive heartbeat a way of evading the portal's controls?

No. A heartbeat is an ordinary authenticated request that resets an inactivity timer — the same thing a human clicking around the portal does. It does not defeat an access control, solve a challenge, or rotate identity to dodge a limit. Those are the hard blocks the parent area stops at. If a portal responds to a normal keep-alive with a challenge, the keeper treats that as a hard block and escalates, exactly as it would for any other anti-bot boundary.

How do I keep the persisted `storage_state` file from becoming a security liability?

Treat it as a live credential, because it is one — anyone holding it can resume the authenticated session. Store it on an access-controlled, encrypted volume, scope it to the service identity that created it, rotate it on the same schedule as the underlying account, and never commit it or log its contents. The keeper deliberately separates the storage_state file from the login credentials so the two rotate independently and a leaked session file can be invalidated without changing the password.

Should every entity read re-authenticate to be safe, rather than sharing one session?

No — that defeats the purpose and invites a rate limit. Re-logging in per entity is slow and is itself the behavior most likely to trip a challenge on a legacy portal. The keeper shares one session across the sweep, cheaply probes it before each read with ensure_alive, and re-authenticates only when the probe shows the session actually died. That gives you the certainty of a per-read check without the cost of a per-read login.