Skip to content
Frankensurf

Read

Lightning mode

Start three tools at once on a page and keep the first complete one.

An ordinary read starts with the cheapest tool and climbs only when it has to. Since v0.33 a read that has no page after 3 seconds starts two racers beside it (smart lightning), so slow reads are already raced by default. Lightning mode starts the read and the two racers at once, for reads where every second matters.

  • The racers are the first free tools on the completeness ladder: by default Jina Reader, which fetches from its own network, and Scrapling, a stealth browser. They start half a second apart, after the read itself, and the read leaves those two tools to them, so no tool reads the page twice.
  • The first complete page wins; the other reads are cancelled.
  • When no page is complete, the more complete of the read’s page and the racers’ best stands; when the read failed, the racers’ best does.
  • An incomplete page then escalates, racing up to four of the tools not tried yet instead of two.
  • Free tools only: lightning never starts a paid tool.
  • A read shaped by a site module is not raced: the module reads the page from its own kind of source.

Measured on 40 sites from the 150-site held-out set, one read at a time, with the order of the two modes shuffled per site: the same 35 sites read either way, the median read fell from 5.0 to 1.9 seconds, and reads that took over 10 seconds normally came back in 2.7 (median). The slowest tenth stayed about the same (14.9 against 15.7 seconds): when the read’s own page is incomplete and the racers are slow, lightning waits for them. With many reads running at once on one machine, the extra browsers compete for its CPU and the gain shrinks.

On a fresh blind set of 150 sites, read from a cloud VM, lightning read 124 (82.7%) at a median of 1.7 seconds, against 118 (78.7%) and 5.9 seconds for ordinary reads four at a time (benchmarks).

Usage

Section titled “Usage”
page = await web.read(url, policy_overrides={"lightning": True})

Set FRANKENSURF_LIGHTNING=1 (in the environment or ~/.config/frankensurf/.env) to turn it on for every read, for example on an MCP server where every answer is waited on. It is a default: a call that sets lightning: false reads normally.

The receipt

Section titled “The receipt”

receipt.hedge describes the race: lightning: true, the racers in providers, the ones that finished in tried, and won: true with the winning racer in provider when a racer’s page was kept. A racer that wins becomes the site’s route hint, so the next read of that site starts with it.