`, and otherwise keeps the block with the most paragraph text. It is a basic fallback; trafilatura is better on unusual layouts. - **Markdown from a hosted reader** (Jina Reader). The link lists before the first real paragraph and after the last one are cut, and the title kept. A page whose text is only a cookie or consent notice counts as incomplete, so the read escalates instead of handing the notice back. --- # Markdown Ask sites for markdown first, so your agent reads clean text. ## Any page as markdown `markdown_output` converts the page Frankensurf got into markdown, whichever tool got it: headings, paragraphs, nested lists, links and images (made absolute), bold and italics, code blocks, tables and quotes. It comes back as `markdown`, next to the plain `text`. Scripts, styles and forms are dropped; menus stay (add `main_content` for the article alone). A page the site already served as markdown is passed through as it is. ```python page = fs.read("https://docs.python.org/3/library/asyncio-task.html", markdown=True) print(page.markdown[:500]) ``` `--markdown-output` on the CLI, `markdown` in the MCP `read` tool. `frankensurf serve` always returns it in Firecrawl's `markdown` field. ## Ask the site for markdown With `prefer_markdown`, the first request asks for `text/markdown`. Sites that support [Markdown for Agents](https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents) answer with markdown directly; everyone else gets the usual HTML route. - No browser and no extra request when the site supports it. - On by default in the MCP `read` tool. - [Jina Reader](/docs/integrations/jina-reader/) returns markdown for sites that don't. ## Usage ```python page = await web.read("https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/", policy_overrides={"prefer_markdown": True}) ``` ```bash frankensurf read https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/ --markdown ``` ```json { "tool": "read", "arguments": { "url": "https://developers.cloudflare.com/fundamentals/reference/markdown-for-agents/" } } ``` ```json title="Response" { "title": "Markdown for Agents", "content_type": "text/markdown; charset=utf-8", "text": "---\ndescription: Cloudflare's Markdown for Agents converts HTML to Markdown at the edge…", "structured": { "format": "markdown" }, "receipt": { "method": "http", "latency_ms": 261 } } ``` ## When to turn it off Markdown keeps links and headings, so it is not always shorter than Frankensurf's extracted text. If you need the page's HTML structure, pass `prefer_markdown: false` (over MCP, in `acquisition_policy`). --- # Page data Get the structured data a page already carries, including the JSON its own frontend fetched. Most pages carry their data in machine-readable form before any text extraction. Frankensurf hands it over as-is, so your agent can read prices, dates or listings without scraping the layout. - `structured.jsonld`: every JSON-LD block the page publishes. - `structured.embedded_json`: JSON embedded for the page's own frontend, such as Next.js's `__NEXT_DATA__`. - `captured_json`: on rendered reads, the JSON responses the page fetched from its own API. - `links`, with `include_links`: every link on the page as `url` and anchor text, in page order and once each (up to `max_links`, 500), so a results page's item URLs come back with it. `--links` on the CLI, `links` in MCP. ## Usage ```python page = await web.read("https://shop.example.com/p/2231", policy_overrides={ "render": True, "capture_json_responses": True, }) offers = [item["data"] for item in page.get("captured_json", {}).get("items", [])] ``` ```bash frankensurf read https://shop.example.com/p/2231 --render --capture-json ``` ```json { "tool": "read", "arguments": { "url": "https://shop.example.com/p/2231", "render": true, "acquisition_policy": { "capture_json_responses": true } } } ``` ```json title="Response (trimmed)" { "structured": { "jsonld": [{ "@type": "Product", "name": "…", "offers": { "price": "…" } }], "embedded_json": [] }, "captured_json": { "items": [{ "url": "https://shop.example.com/api/product/2231", "http_status": 200, "content_type": "application/json", "format": "json", "data": { "title": "Widget", "price": 42 } }], "skipped": 0 } } ``` ## JSON responses When the URL itself returns JSON, `structured` is the parsed body. Newline-delimited JSON becomes `structured.records`. ## Options | Option | Default | Description | | --- | --- | --- | | `capture_json_responses` | `false` | Capture the page's own JSON responses (rendered and signed-in reads). | | `capture_json_max_items` | `20` | Most responses kept. | | `capture_json_max_bytes` | 2 MB | Largest response kept. | Frankensurf returns data, not answers. Deciding which price is current or whether an item is in stock is your agent's job. --- # PDFs Read PDFs as text, with no extra setup. When a URL returns a PDF, Frankensurf extracts its text with [pypdf](https://pypdf.readthedocs.io/). Your agent gets text, not bytes. ## Usage ```python page = await web.read("https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf") ``` ```bash frankensurf read https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf ``` ```json { "tool": "read", "arguments": { "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf" } } ``` ```json title="Response" { "url": "https://www.w3.org/WAI/ER/tests/xhtml/testfiles/resources/pdf/dummy.pdf", "text": "Dummy PDF file", "content_type": "application/pdf; qs=0.001", "structured": { "format": "pdf", "pages": 1 }, "receipt": { "method": "http", "latency_ms": 414 } } ``` ## Options | Option | Default | Description | | --- | --- | --- | | `pdf_max_pages` | `50` | Most pages to extract. Also the most pages a paid tool such as Firecrawl is asked to parse; it bills one credit per page, and a cost cap skips it when the estimate is over. | | `max_bytes` | 40 MB | Largest file accepted. | A PDF longer than `pdf_max_pages` (or than a cost cap pays a paid tool to parse) is cut short, and the result says so. `structured` carries `pages` (read), `pages_total` and `truncated`, and the receipt a short note: ```json "structured": { "format": "pdf", "pages": 10, "pages_total": 48, "truncated": true }, "receipt": { "pdf": { "pages_read": 10, "pages_total": 48, "truncated": true, "note": "PDF truncated: read 10 of 48 pages; raise pdf_max_pages" } } ``` A paid tool returns markdown or HTML, so there the same fields sit under `structured.pdf`. Firecrawl reports the true count as `totalPages`. When it does not, `pages_total` is `null` and a parse that stopped at the page limit counts as truncated. Scanned PDFs without a text layer return no text. OCR is not built in yet. --- # Profiles Sign in once in a Frankensurf browser, then read signed-in pages through any tool that can carry the session. A **profile** is a login that belongs to Frankensurf, not to one browser. You sign in once; after that, a read can carry the session through whichever tool gets the page, and signed-in reads climb the same [escalation ladder](/docs/blocked/escalation/) as everything else. - The session (cookies and local storage) is encrypted on disk. Your agent passes a profile name, never a cookie. - Each profile covers a list of sites. Its cookies go only to those sites, and a profile read of any other site is refused. - A profile pins one fingerprint (user agent, locale, timezone, window size), so the site sees the same browser every time. - Each read saves the refreshed cookies back, so sessions stay current. ## Create a profile 1. Sign in once. ```bash frankensurf profile-login shopping --site example-shop.com ``` A Frankensurf browser window opens. Sign in as you normally would (2FA included), then close the window. Frankensurf never types or clicks here. 2. Check it. ```bash frankensurf profile-list ``` ## Usage ```python page = await web.read("https://www.example-shop.com/orders", policy_overrides={"profile": "shopping"}) ``` ```bash frankensurf read https://www.example-shop.com/orders --profile shopping ``` ```json { "tool": "read", "arguments": { "url": "https://www.example-shop.com/orders", "profile": "shopping" } } ``` `receipt.profile` shows the profile, its version and the tool that carried it. ## Which tools carry a profile | Sharing | Tools | | --- | --- | | `local` (default) | [HTTP](/docs/integrations/http/) with the cookies, [Chromium](/docs/integrations/local-chromium/), [Patchright](/docs/integrations/patchright/) | | `hosted` | adds [Steel](/docs/integrations/steel/), [Browserbase](/docs/integrations/browserbase/), [Hyperbrowser](/docs/integrations/hyperbrowser/), [Browserless](/docs/integrations/browserless/), [Kernel](/docs/integrations/kernel/), [Anchor](/docs/integrations/anchor/) | Choose with `--sharing hosted` on `profile-login`. `hosted` puts the session on that service's servers while it reads. Camoufox is Firefox, so it would contradict a profile's Chromium fingerprint and doesn't carry profiles. Scrape APIs don't carry them either. ## When the session expires A read that lands on a sign-in page fails with `AUTH_REQUIRED`, and `receipt.next_step` names the command that fixes it: `frankensurf profile-login shopping --site example-shop.com`. Signing in again updates the same profile. ## Where it's stored Profiles are AES-GCM files in `~/.local/share/frankensurf/profiles`, readable only by you. The key is `FRANKENSURF_PROFILE_KEY` (32 random bytes, base64) if you set it, which lets you keep it in a secrets manager such as Infisical; otherwise a key file is created at `~/.config/frankensurf/profile.key` with mode 600. Without the key, the profile files are unreadable. Profiles and [identities](/docs/read/signed-in/) are separate. An identity reads through your own running Chrome and never moves its session. A profile is Frankensurf's own copy of a login, so any capable tool can use it. Use an identity when the site must only ever see your real browser. ## Commands | Command | Does | | --- | --- | | `profile-login NAME --site SITE [--site ...] [--sharing local\|hosted]` | Sign in and save, or refresh an existing profile | | `profile-list` | Names, sites, sharing level and cookie counts; never the session | | `profile-delete NAME` | Remove a profile | --- # Signed-in pages Read pages as you, through your own Chrome, without your agent ever seeing a password or cookie. Some pages only exist when you're signed in: orders, saved searches, messages. Frankensurf reads them through a Chrome you sign in to yourself. Your agent passes a name, never a password, cookie or profile. For most sites a [profile](/docs/read/profiles/) is simpler: sign in once in a Frankensurf browser and any capable tool can carry the session. Identities are for sites that must only ever see your real Chrome. - Each saved login is an **identity**, tied to one Chrome profile and a list of sites. - Pages run normally, scripts included, so single-page apps like dashboards and inboxes render. - If anything about the browser looks wrong, the read stops. It never falls back to a logged-out read. ## Set up an identity 1. Start a dedicated Chrome with remote debugging on a loopback port and its own profile. Sign in to the site yourself. 2. Register that Chrome. ```bash frankensurf executor-enroll my-chrome \ --cdp-url http://127.0.0.1:9222 \ --user-data-dir /home/you/.frankensurf/profiles/main \ --profile-directory Default ``` 3. Register the identity: which sites it covers, and how to tell you're signed in. ```bash frankensurf identity-enroll shop-me \ --executor-id my-chrome --domain www.example-shop.com \ --auth-url https://www.example-shop.com/account \ --authenticated-selector '#account-menu' ``` `scripts/prepare_local_identity.py` prints a reviewable plan for a dedicated profile, and with `--launch` opens it for you to sign in. ## Usage ```python page = await web.read("https://www.example-shop.com/orders", policy_overrides={"identity": "shop-me"}) ``` ```bash frankensurf read https://www.example-shop.com/orders --identity shop-me frankensurf identity-status shop-me ``` ```json { "tool": "read", "arguments": { "url": "https://www.example-shop.com/orders", "identity": "shop-me" } } ``` ## What it guarantees - Every top-level navigation, including redirects, must stay on the identity's sites. Popups are closed. - Frankensurf never clicks or types in a signed-in tab during a read. - One process uses an identity at a time; your own browsing in other tabs is unaffected. - Output URLs drop credential parameters such as `access_token`. Remove an identity with `frankensurf identity-revoke shop-me`. Identities run on your machine only (`LOCAL_ONLY`). Stored secrets are in private files (mode 600) but not yet encrypted at rest. --- # Site modules Save what your agent learns about a site, as data, and reuse it on every read. When your agent works out how a site really works, it can save that as a **site module** instead of writing a throwaway parser. Typical lessons: - the search URL only takes `q`; - results sit as JSON inside the page; - listing URLs live in JSON-LD and need pairing with the cards. Every later read of that site comes back shaped: `items` with the fields you named, the next page, and checks that tell you when the site changes. A module is **data, never code**, and lives in your own state folder. Frankensurf ships none; site knowledge stays with whoever learned it. ## Items with no module at all Pass `auto_items` (MCP: `items=True`) on any read. If the page carries a list of items in its own data, the result gets `items`, found the same way as below and with no extra read, plus `auto_module`: the module that found them. Save it (`site_modules` `put`) and every later read of that kind of page is shaped without asking. ```python page = await web.read(url, policy_overrides={"auto_items": True}) page["items"][:3]; page["receipt"]["auto_items"] # {"found", "count", "source"} ``` ## Let Frankensurf draft one Most listing pages carry their rows as data for their own frontend: JSON-LD, JSON in a script tag (`__NEXT_DATA__` and friends), or JSON the page fetches while it renders. When none of that holds the rows, they are usually repeated HTML cards. `module discover` reads a page, looks in all four places, and drafts a module from the best list of items it finds (name, link, price, image). Each draft is validated and run against that same page, so you see the item count and sample rows before you save anything. ```python found = await web.discover_module("https://www.example.com/search?q=lamp") best = found["drafts"][0] # module, count, fields, sample await web.discover_module("https://www.example.com/search?q=lamp", save=True) ``` ```bash frankensurf module discover "https://www.example.com/search?q=lamp" --save ``` ```json { "tool": "discover_site_module", "arguments": { "url": "https://www.example.com/search?q=lamp", "save": true } } ``` After saving, every read of a matching URL (another query, the same path) comes back with `items`. Check the sample first: a draft can pick the wrong list on a page with several, and nothing here knows a site. On 16 listing pages it had never seen, it drafted the right list on 8, the wrong one on 1, and nothing on 7 (mostly pages it could not read). Keep the drafts that are right, and edit or write the rest by hand. Start from a home page instead, with a query, when you don't know the site's search URL. Frankensurf finds the search itself: schema.org `SearchAction` first, then the page's search form, then, for a search box run by script, it types the query once into the visible box in a local browser, presses Enter and keeps the URL it lands on. Nothing else is typed or clicked. ```bash frankensurf module discover https://www.example.com/ --query "desk lamp" --save ``` On 14 home pages it had never seen, it found the search on 10 (three of the others refused the read) and drafted the right results on 6. When the page was a search, the draft also gets a `search` template built from its URL: the parameter that held the search words (`?q=lamp`), or the path segment after `/q/`, `/search/` or `/tag/`. Everything else in the URL stays as it was. So one saved search page becomes any search: ```python page = await web.read_template("auto-example-com", "search", {"query": "desk lamp"}) ``` ## How far to trust the items Every drafted module, every module read and every `auto_items` result carries an `items_quality` check against the page itself: - `linked`: items whose link stays on the site (or the site it redirected to) and looks like an item page, not this page, a help page or a filter of this listing; - `named`: items whose name appears in the page's text; - `priced`: items with a positive price; - `distinct`: items that are not all the same link. A `grade` (`good`, `partial`, `poor`) sums it up. Discovery drafts several candidates and ranks them by this check before size; poor lists are dropped. On 16 unseen listing pages it drafted the right items on 10 (8 before the check), the wrong ones on 1, and none on 5 that refused the read. When a saved module comes back `poor` on a page that read fine, the site probably changed. The receipt then carries `module.repair_draft`: a new version drafted from that same page, keeping your templates and pagination. Nothing is saved until you accept it with `site_modules` `put`. ## What a module holds ```json { "id": "example.search", "version": "1", "match": { "origin": "https://www.example.com", "path_pattern": "/search" }, "templates": { "search": { "url": "https://www.example.com/search?q={query}", "params": { "query": "required" } } }, "sources": { "hits": { "kind": "embedded_json", "decode": "next_flight", "marker": "\"hits\":" } }, "items": { "from": "hits", "fields": { "title": "ObjectTitle", "url": { "path": "ITEM_URL", "type": "url" }, "price": { "path": "ObjectPrice", "type": "number" } } }, "assertions": { "schema": "frankensurf.workload-assertions/v1", "checks": [ { "path": "items", "operator": "minimum_count", "value": 5 }, { "path": "matching_query", "operator": "minimum_count", "value": 3 } ] }, "notes": "Search takes q; searchTerm is silently ignored." } ``` | Part | What it does | | --- | --- | | `match` | `origin` and a `path_pattern` (a regular expression over the URL path), like a route recipe. Optional `query_keys`. | | `templates` | URLs with `{params}`, each `"required"` or `{default, encoding: "query" \| "path"}`. They always stay on the module's origin. | | `readiness` | `selector` to wait for (bounded), `settle_ms`. | | `sources` | Where rows live: `jsonld` (`type`, `path`), `embedded_json` (`selector`, `marker`, `path`, optional `decode: "next_flight"` for Next.js pages), `captured_json` (JSON the page fetched: `url_contains`, `path`), or `html` (`item_selector` and `fields`, each a `selector` and optional `attribute` or list of attributes tried in order). | | `items` | A field map over one source: `name: "path"` or `{path, type}` with `type` `auto`, `text`, `number` or `url`. `join` fills blanks from a second source, by `position` or by `{left, right}` key. `limit`, default 200. | | `pagination` | `{param, start, step, max_pages}` or `{next_selector}`. The read returns `next_url`. | | `invalid` | Markers of the site's error or empty-feed page: `final_url_contains`, `title`, `text`. Such a page fails as `NOT_FOUND` instead of climbing to stronger tools. | | `assertions` | A `frankensurf.workload-assertions/v1` set over `{items, count, first, matching_query}`. Operators are `nonempty`, `equals` and `minimum_count`. | | `policy_defaults` | Operational settings, such as `scroll_screens` or `settle_ms`. | **What a module can never do:** name an identity, pick or allow providers, allow paid tools, or reach another origin. Any setting you pass on the call wins over the module's. ## Using modules - **Saved modules:** an enabled one that matches the URL shapes the read by default. - **One read:** name a module with `module`, run an unsaved one with `module_override`, or turn modules off with `module=False` (`none` on the CLI and MCP). - **What comes back:** `items` and, with pagination, `next_url`. The receipt gains `module`: its `id`, `version`, `sha256`, how it was chosen, the item count and the assertion results (`status`: `passed`, `failed` or `invalid`). Raw page data is unchanged. - **Completeness:** a page whose module assertions pass counts as complete, so a site that keeps its results in JSON isn't escalated for having few links. ```python web.site_modules.put(module) # save or replace (bump version) page = await web.read_template("example.search", "search", {"query": "desk lamp"}) for item in page["items"]: print(item["title"], item["price"], item["url"]) print(page["receipt"]["module"]["status"]) # passed / failed / invalid await web.read(url, module=False) # without modules web.site_modules.enable("example.search", False) # or disable it ``` ```bash frankensurf module add example-search.json frankensurf module list frankensurf read-template example.search search --param query="desk lamp" frankensurf read https://www.example.com/search?q=lamp --module none frankensurf module disable example.search ``` ```json { "tool": "site_modules", "arguments": { "action": "put", "module": { "id": "example.search", "version": "1", "...": "..." } } } { "tool": "read_template", "arguments": { "module": "example.search", "template": "search", "params": { "query": "desk lamp" } } } ``` ## When a site changes When a module's assertions start failing, `receipt.module.status` says `failed` (or `invalid`), `receipt.module.repair` says what to do, and the raw page is still there. Your agent can either: - **Save a fix directly.** Put the module again under a new `version`. Fine for your own modules. - **Propose a fix through repair.** Use this when changes should be reviewed. The agent drafts the next version (same `id` and origin, new `version`) and proposes it with the failing read's `trace_id`. Frankensurf then: 1. checks it with the **old version's assertions**, so a fix can't pass by weakening the checks; 2. runs it over the page the failing read kept; 3. runs it over a fresh, independent read; 4. records the proposal and its validation. Nothing changes yet. The owner promotes it with `repair-promote`. Promotion re-checks the kept page, refuses if the module changed in the meantime, saves the new version and records it in the repair history. `repair-disable` rolls back to the version it replaced. ```python proposal = await web.propose_module_repair(failing_trace_id, next_version) # proposal["status"] == "proposal_ready" once the kept page and a fresh read pass web.promote_repair(proposal["proposal"]["id"], proposal["proposal"]["sha256"]) # owner web.disable_repair(proposal["proposal"]["id"], "rolled back after review") # undo ``` ```bash frankensurf module repair TRACE_ID example-search-v2.json frankensurf repair-promote PROPOSAL_ID --proposal-sha256 SHA256 frankensurf repair-disable PROPOSAL_ID --disable-reason "rolled back after review" ``` ```json { "tool": "repair_site_module", "arguments": { "trace_id": "…", "module": { "id": "example.search", "version": "2", "...": "..." } } } ``` Agents can propose, but only the owner promotes, from the CLI or Python. --- # Structured fields Pass a JSON Schema and get its fields filled from the data the page publishes, with a climb to a stronger tool while a required field is missing. Most of the time an agent wants a few fields from a page, not the page: a product's name and price, an article's headline and date, a job's company and location. Pass a JSON Schema and the result gets `extracted`, with each property filled from what the page publishes for machines. ```python schema = { "type": "object", "properties": { "name": {"type": "string"}, "price": {"type": "number"}, "currency": {"type": "string"}, "image": {"type": "string"}, }, "required": ["name", "price"], } page = await web.read(url, policy_overrides={"extract_schema": schema}) page["extracted"]["data"] # {"name": "Brass desk lamp", "price": 89.95, "currency": "AUD", ...} page["extracted"]["missing"] # required properties not found: [] page["extracted"]["sources"] # {"price": "jsonld:product.offers.price", ...} ``` ```bash frankensurf read https://shop.example.com/item/42 --schema product.json ``` ```json { "tool": "read", "arguments": { "url": "https://shop.example.com/item/42", "schema": { "type": "object", "properties": { "name": { "type": "string" }, "price": { "type": "number" } }, "required": ["name", "price"] } } } ``` ## Where the values come from No model is used. Values come from what pages publish for search engines and link previews: - **JSON-LD** (schema.org): `Product`, `Article`, `JobPosting`, `Event` and the rest, including `@graph` blocks. - **Microdata** (`itemprop`). - **Meta tags:** Open Graph (`og:title`, `og:image`), `article:published_time`, `product:price:amount` and the like. - **The page's own JSON**, such as a framework's embedded state, searched by key name. A property is matched by its name and common synonyms: `price` finds `offers.price`, `lowPrice` and `product:price:amount`; `published` finds `datePublished` and `article:published_time`; `name` finds `headline` and `og:title`. Values are converted to the property's type, so `"1,299.00"`, `"1.249,50 €"` and `"1 249"` all become numbers. Name your own paths with `x-paths`: ```json { "cheapest": { "type": "number", "x-paths": ["offers.lowPrice", "meta:product:price:amount"] } } ``` Give the schema a `title` (or `x-type`) such as `"Product"` and the matching JSON-LD object is preferred when a page has several. The filled object is validated against the schema; `valid` and `errors` say how it went. ## Missing fields make the read try harder A page that loaded but lacks a required property counts as incomplete, the same as a search page with no results. An automatic read then climbs to a stronger tool, and keeps the page that fills the most properties. `receipt.extracted` records `found`, `of` and `missing`. A field the page never publishes in its data stays missing. For those, read the page's text (`main_content` helps) and let your agent pick the value out. --- # Command line Every frankensurf command and option. ```bash frankensurf [targets] [options] ``` Output is JSON. The raw page body is left out unless you pass `--raw`. State goes to `--state` (default `./state`). ## Commands | Command | Targets | What it does | | --- | --- | --- | | `setup` | none | Download Chromium and the free stealth browsers (`--no-stealth` skips them), then check each one starts and reads a page: `works_here` in the result. | | `read` | one URL | Read a page. `--module ID` uses a site module; `--module none` turns them off. | | `read-template` | module ID, template | Read a URL built from a site module's template; `--param NAME=VALUE` per parameter. | | `module` | `list`, `show ID`, `add FILE`, `enable ID`, `disable ID`, `rm ID`, `repair TRACE_ID FILE`, `discover URL [--query TEXT] [--save]` | Manage [site modules](/docs/read/site-modules/); `repair` proposes a fixed version for owner promotion with `repair-promote`; `discover` drafts one from a page's own data. | | `watch-sites` | site URLs | New pages on those sites from their feeds and sitemaps; `--since DATE`, `--read-new`. | | `skill` | `install` or `uninstall` (`--skill-dir`), or `show` | Install, remove or print the agent skill (SKILL.md). | | `plugins` | `list`, `trust KIND ID DISTRIBUTION`, `untrust KIND ID`, `disable KIND ID`, `enable KIND ID` | Third-party [plugins](/docs/more/plugins/): what loaded, what was turned away and why, and the trust file. | | `egress` | `check` | Whether your direct connection and each [second connection](/docs/blocked/egress/) works, and where it comes out (address, country, WARP). | | `extract` | one URL | Read through `--adapter` (default `html`). | | `batch` | URLs | Read several pages. | | `search` | query words | Search. `--source` picks one source; `--limit` (default 10). | | `watch` | one URL | One poll for new links; `--link-pattern RE`. | | `paginate` | one URL | Follow pages; needs `--adapter`; `--max-pages`, `--continuation-adapter`. | | `images` | image URLs | Download and check images. | | `trace` | one trace ID | Show how a result was fetched. | | `import` | one URL | Store a capture you made: `--evidence-file`, `--observed-at`, `--content-type`. | | `do` | none | A typed browser action from `--intent-file`; needs `--identity`, `--grant-action`, `--action-origin`. | | `repair` | one trace ID | Diagnose a retained extract failure; `--repair-policy-file`. | | `repair-promote`, `repair-disable` | one proposal or overlay ID | Activate (`--proposal-sha256`) or disable (`--disable-reason`) a repair overlay. | | `executor-enroll` | executor ID | Register a Chrome: `--cdp-url`, `--user-data-dir`, `--profile-directory`, `--network-context`, `--geography`. | | `identity-enroll` | identity ID | Register a login: `--executor-id`, `--domain`, `--image-domain`, `--auth-url`, `--authenticated-selector`, `--login-selector`, `--allow-action`. | | `identity-status` | optional identity ID | Health of your identities. | | `identity-revoke` | identity ID | Remove an identity. | | `bot-auth-init` | none | Create a Web Bot Auth key. | | `bot-auth-directory` | none | Print the public key directory to serve. | ## Read options These map to [WebPolicy](/docs/reference/settings/) fields and work on `read`, `extract`, `batch`, `watch` and `paginate`. | Option | Field | | --- | --- | | `--provider ID` | `provider` | | `--provider-candidate ID` (repeat) | `provider_candidates` | | `--markdown` | `prefer_markdown` | | `--render` | `render` | | `--handoff` | `allow_handoff` | | `--lightning` | `lightning` | | `--real-browser` | `allow_real_browser` | | `--paid`, `--max-cost USD` | `allow_paid_fallbacks`, `max_cost_usd` | | `--capture-json` | `capture_json_responses` | | `--images`, `--max-images N` | `include_images`, `max_images` | | `--card-images`, `--scroll-screens N` | `card_images`, `scroll_screens` | | `--links`, `--max-links N` | `include_links`, `max_links` | | `--markdown-output` | `markdown_output` | `map URL` (`--query`, `--limit`) and `crawl URL` (`--limit`, `--max-depth`, `--include-path`, `--exclude-path`, `--entire-domain`, `--ignore-robots`): see [Map and crawl](/docs/crawl/). `ask "question"` (`--model`): agent mode, see [Agent mode](/docs/agent-mode/). `serve --model` turns on the summary, json and extract endpoints. | `--expect WORD` (repeat) | `expect_terms` | | `--freshness now\|hour\|day\|cached` | `freshness` | | `--timeout S` | `timeout_seconds` | | `--wait-selector CSS` | `wait_selector` | | `--content-ready-selector CSS`, `--content-ready-timeout S` | `content_ready_selector`, `content_ready_timeout_seconds` | | `--agent-task TEXT` | `browser_agent_task` | | `--identity ID` | `identity` | | `--max-pages N` | `max_pages` | | `--provider-max-attempts-per-candidate N`, `--provider-retry-delay S`, `--provider-retry-failure CODE` | retry fields | Search adds `--search-source-candidate`, `--search-source-allow`, `--search-source-prefer` (each repeatable), `--search-max-attempts`, `--search-source-timeout` (0 means the operation timeout) and `--search-url`. `--paid` allows paid tools whose keys are set (`allow_paid_fallbacks`) and `--max-cost USD` caps what one operation may spend (`max_cost_usd`). Fields without a flag are set from Python or MCP. ## Other options | Option | What it does | | --- | --- | | `--state DIR` | State directory (default `state`). | | `--steel-url URL` | A self-hosted Steel. | | `--identity-registry PATH` | Identity registry (default `FRANKENSURF_IDENTITIES` or `~/.frankensurf/identities.json`). | | `--raw` | Include the raw page body. | --- # Environment and files Every environment variable and file Frankensurf reads. ## Environment variables | Variable | What it sets | | --- | --- | | `FRANKENSURF_STATE` | State directory for the MCP server. | | `FRANKENSURF_PLUGINS` | The plugin trust file (default `~/.config/frankensurf/plugins.json`); see [Writing a plugin](/docs/more/plugins/#the-trust-file). | | `FRANKENSURF_STEEL_URL` | A self-hosted Steel for the `steel` provider. | | `FRANKENSURF_SEARCH_URL` | A SearXNG at another loopback address (MCP server). Not needed for the one `frankensurf install search` starts, at `http://127.0.0.1:8088`. | | `FRANKENSURF_GOOGLE_CSE_CX` | The search engine ID for [Google Programmable Search](/docs/integrations/google-cse/), beside `GOOGLE_CSE_API_KEY`. | | `FRANKENSURF_SEARCH_AUTOSTART` | `0`: search never starts the installed local SearXNG on its own (by default it does, once per process, when nothing answers on `127.0.0.1:8088`). | | `FRANKENSURF_IDENTITIES` | Identity registry (default `~/.frankensurf/identities.json`). | | `FRANKENSURF_PROVIDER_PYTHON` | Python for the isolated stealth browsers. | | `FRANKENSURF_CRAWL4AI_DIR`, `FRANKENSURF_CRAWL4AI_PYTHON`, `FRANKENSURF_CRAWL4AI_BROWSERS_PATH`, `FRANKENSURF_CRAWL4AI_HEALTH` | The isolated Crawl4AI install. | | `FRANKENSURF_BROWSER_USE_PYTHON`, `FRANKENSURF_BROWSER_USE_BROWSER`, `FRANKENSURF_BROWSER_USE_MODEL_FACTORY`, `FRANKENSURF_BROWSER_USE_BILLING`, `FRANKENSURF_BROWSER_USE_BINDING_REVISION` | The Browser Use agent. | | `FRANKENSURF_SIGNATURE_AGENT`, `FRANKENSURF_BOT_AUTH_KEY_FILE` | [Signed requests](/docs/blocked/signed-requests/). | | `FRANKENSURF_HANDOFF_PROFILE`, `FRANKENSURF_HANDOFF_CDP_URL` | [Handoff](/docs/blocked/handoff/). | | `FRANKENSURF_REAL_BROWSER`, `FRANKENSURF_REAL_BROWSER_PATH`, `FRANKENSURF_REAL_BROWSER_PROFILE`, `FRANKENSURF_REAL_BROWSER_CDP`, `FRANKENSURF_REAL_BROWSER_WINDOW` | [Your real browser](/docs/blocked/real-browser/): `1` turns it on for every read. | | `FRANKENSURF_EGRESS` | [Your other connections](/docs/blocked/egress/), `name=url`, for walled reads. | | `FRANKENSURF_LIGHTNING` | `1` turns [lightning mode](/docs/read/lightning/) on for every read. Both switches also work in `~/.config/frankensurf/.env`, and both are defaults: a call that sets the field itself wins. | | `FRANKENSURF__API_KEY` and rate settings | Paid services. See [Keys and costs](/docs/blocked/paid-tools/). | ## Files | Path | What it holds | | --- | --- | | `./state/` (or `--state`) | Cache, evidence, traces, route memory, `origin-pacing.json`, `watches/`. | | `~/.config/frankensurf/.env` | Keys and settings as `FRANKENSURF_…=…` lines. | | `~/.config/frankensurf/services.json` | Keys under `api_keys`. | | `~/.config/frankensurf/web-bot-auth.jwk` | Your Web Bot Auth private key. | | `~/.config/frankensurf/browser-agent.json` | Browser Use model configuration. | | `~/.config/frankensurf/plugins.json` | Which installed third-party plugins to trust. | | `~/.frankensurf/identities.json` | Registered executors and identities. | | `~/.local/share/frankensurf/handoff-profile/` | The handoff browser profile. | | `~/.local/share/frankensurf/crawl4ai-venv/` | The Crawl4AI environment. | Keep everything under `~/.config/frankensurf` readable only by you. --- # Errors What each failure code means and what to try next. A failed read has `receipt.status: "failed"` and `receipt.failure` with a code and a message. `receipt.attempts` shows what each provider hit on the way. ## Walls | Code | What happened | Next | | --- | --- | --- | | `BLOCKED` | The site refused (403, or an empty block response). | Climbs automatically; starts a cool-down. Allow paid fallbacks or handoff. | | `CAPTCHA` | A challenge page. | Same as `BLOCKED`. `receipt.next_step` may suggest handoff. | | `RATE_LIMITED` | Too many requests (429). | Wait; pacing and the cool-down handle it. | | `AUTH_REQUIRED`, `AUTH_EXPIRED` | The page needs a sign-in. Stops the climb. | Use an [identity](/docs/read/signed-in/) or handoff. | ## Page problems | Code | What happened | Next | | --- | --- | --- | | `VISUAL_REQUIRED` | Plain HTTP got an app shell or an unrendered template. | Climbs to a browser automatically. | | `EMPTY_PAGE` | A browser rendered almost no text, or any tool, named or not, got a page with nothing on it (no text, page data, links or forms). | Climbs automatically. | | `NOT_FOUND` | 404 or 410. Stops the climb unless an earlier attempt hit a wall. | A missing page does not prove a sale. | | `CONTENT_MISMATCH` | The page is not what was asked for, for example a search that was redirected to the site's home page. | Climbs automatically; check the URL. | | `SCHEMA_CHANGED` | The response did not have the expected shape. | Check the adapter, or the service changed. | | `TIMEOUT` | Too slow. | Raise `timeout_seconds`. | | `LIMIT_EXCEEDED` | Too big. | Raise `max_bytes`. | | `INVALID_URL` | Not a valid URL. | Fix it. | | `INVALID_IMAGE` | One image did not decode. | Only that image fails. | ## Providers, policy and cost | Code | What happened | | --- | --- | | `PROVIDER_DOWN` | The provider or site had a server error. Free providers retry once. | | `PROVIDER_UNAVAILABLE` | Not installed or not configured (no key, no display), including a browser build Playwright expects but can't find: run `frankensurf install chromium`. | | `PLUGIN_DISABLED` | The provider is disabled in the catalogue. | | `POLICY_DENIED` | Your settings allow no provider for this. | | `BUDGET_EXHAUSTED` | The next call could pass `max_cost_usd`. | | `EXPLICIT_POLICY_CONFLICT` | A whole `WebPolicy` conflicts with the site's route seed; pass `policy_overrides` instead. | | `ROUTE_SCOPE_REQUIRED` | That provider only runs through a route seed for one site. | | `SEARCH_UNAVAILABLE` | Every search source failed. | | `DISCOVERY_UNAVAILABLE`, `PLUGIN_NOT_TRUSTED`, `PLUGIN_LOAD_REJECTED` | A third-party plugin could not be discovered, is not trusted in `plugins.json`, or failed to load. Shown in `inspect_plugins()`. | | `EXECUTION_OUTCOME_UNKNOWN` | A browser action failed part-way; its idempotency key is fenced from replay. | | `UNKNOWN` | Anything else. Look at the trace. | ## Identities | Code | What happened | | --- | --- | | `IDENTITY_REQUIRED` | Your Chrome was asked for without naming an identity. | | `IDENTITY_UNKNOWN`, `IDENTITY_REVOKED` | Not registered, or removed. | | `IDENTITY_EXECUTOR_OFFLINE`, `IDENTITY_EXECUTOR_UNKNOWN`, `IDENTITY_EXECUTOR_DISABLED` | The registered Chrome is not available. | | `IDENTITY_PROFILE_MISMATCH`, `IDENTITY_CHANGED` | Chrome is not running the registered profile. | | `IDENTITY_IN_USE` | Another process holds the identity. | | `IDENTITY_REAUTH_REQUIRED` | You have been signed out; sign in again. | | `IDENTITY_OPERATOR_HANDOFF_REQUIRED` | A visible snapshot needs you to open exactly one matching page, or its region is missing. | | `IDENTITY_DOMAIN_DENIED`, `IDENTITY_PROVIDER_DENIED`, `IDENTITY_ACTION_DENIED`, `IDENTITY_POLICY_DENIED` | Not allowed for that site, provider or action. | | `IDENTITY_AUTHORITY_UNSUPPORTED` | Only `LOCAL_ONLY` runs today. | | `IDENTITY_CONFIG_INVALID`, `IDENTITY_REGISTRY_INVALID`, `IDENTITY_STORE_UNSAFE` | Bad settings or file permissions. | | `VAULT_REFERENCE_INVALID`, `VAULT_REFERENCE_MISSING`, `VAULT_VALUE_INVALID` | A stored secret is missing or unreadable. | --- # MCP tools The tools frankensurf-mcp gives your agent, and their arguments. `frankensurf-mcp` speaks MCP over stdio. To connect it, see [Add to your agent](/docs/agents/). | Tool | Arguments | Returns | | --- | --- | --- | | `read` | `url`, `provider`, `render`, `include_images`, `freshness`, `identity`, `allow_handoff`, `profile`, `try_harder_than`, `expect_terms`, `card_images`, `links`, `markdown`, `allow_archive`, `items`, `main_content`, `lightning`, `allow_real_browser`, `acquisition_policy` | The page and its receipt. `lightning` starts the read and two other tools at once ([Lightning mode](/docs/read/lightning/)); `allow_real_browser` lets your Chrome or Edge try a walled page ([Your real browser](/docs/blocked/real-browser/)). `main_content` returns the article without menus ([Main content](/docs/read/main-content/)). Asks for markdown by default. `try_harder_than` takes an earlier read's `trace_id` and skips every tool it used. `expect_terms` are words results should mention; a results page that mentions none is flagged `off_query`. `card_images` adds `cards` (link, title, thumbnail); `links` adds every link on the page (url, anchor text); `markdown` adds the page as markdown. `module` names a [site module](/docs/read/site-modules/) (`none` turns them off); `module_override` runs an unsaved one. | | `read_template` | `module`, `template`, `params`, `expect_terms`, `card_images`, `acquisition_policy` | A read of a URL built from a saved module's template, shaped by that module: `items`, `next_url`, `receipt.module`. | | `site_modules` | `action` (`list`, `get`, `put`, `enable`, `disable`, `delete`), `module_id`, `module` | Save and manage [site modules](/docs/read/site-modules/). | | `watch_sites` | `sites`, `since`, `read_new`, `max_new_per_site`, `acquisition_policy` | New pages on many sites from their feeds and sitemaps, polled with conditional requests; `read_new` reads them too. See [Watch](/docs/watch/). | | `batch_template` | `module`, `template`, `params_list`, `acquisition_policy` | One saved template (say, a search) for many parameter sets as one paced batch: each result's `params`, `items`, `next_url` and receipt status, without page text. | | `discover_site_module` | `url`, `save`, `module_id`, `query` | Up to three validated module drafts from a listing page's JSON-LD, script JSON, fetched JSON or repeated HTML cards, each with its item count and sample rows. `save` saves the best. | | `repair_site_module` | `trace_id`, `module` | Propose the next version of a module that failed on a read. Validated against the kept page and a fresh read with the old version's assertions; only the owner promotes it. | | `map_site` | `url`, `search`, `limit`, `include_subdomains` | A site's URLs from its sitemaps and links, without reading them ([Map and crawl](/docs/crawl/)). | | `crawl` | `url`, `limit`, `max_depth`, `include_paths`, `exclude_paths`, `entire_domain`, `acquisition_policy` | Read a section of a site as markdown, breadth first, paced per site. | | `trace` | `trace_id` | The full record of an earlier read: tools tried, failures, timings, completeness steps. No page content. | | `extract` | `url`, `adapter`, `include_images`, `identity`, `provider`, `freshness`, `acquisition_policy` | The page through an adapter: `html` (default), `json` or `rss`. | | `batch` | `urls`, `provider`, `adapter`, `identity`, `include_images`, `freshness` | One result per URL. | | `search` | `query`, `source`, `limit`, `site`, `exclude_domains`, `recency`, `region`, `vertical`, `mode`, `identity`, `provider`, `acquisition_policy` | Attributed results. Options are applied natively and filtered after; `vertical` picks news, reference, discussions, qa, code, papers or books (inferred from the query when omitted); `mode="merge"` fuses every eligible source. See [Search](/docs/search/). | | `paginate` | `url`, `adapter`, `continuation_adapter`, `acquisition_policy` | Every page with an aggregate receipt. | | `images` | `urls`, `identity`, `max_images`, `provider` | Checked images. | | `identity_status` | `identity` | Health of your identities. | | `import_evidence` | `content`, `url`, `observed_at`, `content_type`, `image_urls` | A stored capture. | | `import_image_evidence` | `path`, `url`, `observed_at` | A stored image capture. | | `do` | `intent`, `identity`, `action_classes`, … | A typed browser action. See [Act](/docs/act/). | | `repair` | `trace_id`, `repair_policy` | A diagnosis of a failed extract. | `acquisition_policy` takes any [setting](/docs/reference/settings/), for example `{"allow_paid_fallbacks": true, "max_cost_usd": 0.05}`. JSON `null`, `false` and `0` count as set. MCP results leave out the raw page body. Pass `acquisition_policy: {"prefer_markdown": false}` to `read` when you need the HTML structure instead of markdown. Identities and keys are set up from the command line, never through MCP. `watch` is available from Python and the CLI. --- # Python The Runtime class, its methods and what they return. ```python from frankensurf import Runtime, WebPolicy ``` ## Runtime Open it with `async with`. It owns the state directory: cache, evidence, traces, route memory and pacing. ```python Runtime( state_dir="state", # cache, evidence, traces, route memory steel_api_url=None, # a self-hosted Steel, or FRANKENSURF_STEEL_URL local_cdp_url=None, # used only through a registered identity concurrency=4, # pages at once in a batch per_domain=1, # pages at once per site in a batch domain_delay=0.25, # seconds between batch requests to one site identity_registry=None, # or FRANKENSURF_IDENTITIES plugin_config_path=None, # or ~/.config/frankensurf/plugins.json ) ``` ## Methods Every read-like method takes `policy` (a whole `WebPolicy`) or, preferably, `policy_overrides` (a dict of only the fields you mean). See [WebPolicy](/docs/reference/settings/). | Method | Returns | What it does | | --- | --- | --- | | `await read(url, policy=None, provider=None, adapter=None, *, policy_overrides=None)` | result | Read a page. | | `await extract(url, adapter="html", policy=None, provider=None, *, policy_overrides=None)` | result | Read through an explicit adapter: `html`, `json` or `rss`. | | `await batch(urls, policy=None, adapter=None)` | list of results | Several pages, in order. Identity batches run one at a time. | | `await search(query, source=None, limit=10, engine_config=None, policy=None)` | search result | Search with fallback; a named `source` never falls back. | | `await watch(url, *, link_pattern=None, policy_overrides=None, state_key=None, max_seen=5000)` | `{"new": [...], "first_poll": bool, "receipt": ...}` | One poll for links not seen before. | | `await paginate(url, adapter, policy=None, *, continuation_adapter=None, policy_overrides=None)` | pages and aggregate receipt | Follow a bounded run of pages. | | `await download_images(urls, policy=None)` | list of images | Download and check images. | | `await do(intent, policy=None, provider=None)` | action result | A typed browser action. See [Browser actions](/docs/act/). | | `trace(trace_id)` | dict | The full record of one request. | | `capabilities(domain=None)` | list | Success and speed per site and provider, from traces. | | `identity_status(identity_id=None)` | dict | Health of your identities, without secrets or paths. | | `import_evidence(content, url, observed_at, …)` | result | Store something you captured yourself, with its real time. | | `inspect_plugins()` | dict | The frozen plugin catalogue. | | `await repair(trace_id, policy=None)` | dict | Diagnose one retained public extract failure. | ## Results ```python { "url": "https://example.com/", "title": "Example Domain", "text": "…", "content": "…", # raw body "content_type": "text/html", "headers": {…}, # never cookies "structured": {"jsonld": [], "embedded_json": []}, "image_urls": […], "images": […], # with include_images=True "field_status": {"availability": "unknown", "transaction_price": "unknown", "content": "observed"}, "receipt": {…}, } ``` See [Reading pages](/docs/read/) for every field and [Receipts](/docs/reference/receipts/) for the receipt. ## Failures A failed operation returns a result whose `receipt.status` is `"failed"` and whose `receipt.failure` holds `{"code", "message"}`. Invalid arguments and invalid policies raise `ValueError` straight away. See [Error codes](/docs/reference/errors/). --- # Receipts How every result shows how it was fetched, when, at what cost, and what is still unknown. Every result carries a receipt. It says how the page was fetched, when, what it cost and what Frankensurf could not confirm. Raw content stays on your machine, linked to the receipt by hash. ## A receipt ```json { "trace_id": "6f1c…", "status": "observed", "operation": "read", "method": "local", "observed_at": "2026-10-06T03:14:07Z", "freshness_seconds": 0, "cache_hit": false, "requested_url": "https://example.com", "final_url": "https://example.com/", "http_status": 200, "latency_ms": 2267, "cost_usd": 0, "evidence": [{"path": "state/evidence/2a8c….html", "sha256": "2a8c…", "bytes": 23}], "attempts": [ {"provider": "http", "status": "failed", "failure": "VISUAL_REQUIRED", "latency_ms": 90}, {"provider": "local", "status": "observed", "latency_ms": 2176} ], "routing": {"provider_plan": { "ordered": ["http", "local", "camoufox", "scrapling", "scrapling_http", "crawl4ai", "jina_reader"], "basis": "registration order; insufficient exact-scope evidence" }} } ``` Trimmed from a real read. `routing.provider_plan.ordered` is the route this read planned, after skipping anything unavailable. - **`attempts`** lists every provider tried, in order, with its outcome, time and cost. - **`evidence`** points to saved copies, named by their SHA256 hash, under the state directory. - **`requested_url`** and **`final_url`** stay separate, so redirects show. - **`cost_usd`** is the sum of measured provider costs. If any attempt's cost is unknown, the total is `null`, never a guess. - **`next_step`** appears when a read stopped at a wall a person could clear (it names `handoff`), or when a results page doesn't mention the query (`reason: "off_query"`: the search URL is probably wrong). - **`module`** names the [site module](/docs/read/site-modules/) that shaped the read: `id`, `version`, `sha256`, `source` (`matched`, `named` or `override`), `items`, `next_url`, and `assertions` with `status` (`passed`, `failed`, or `invalid` when its markers matched the site's error page). - **`completeness`** records the structure check: page kind, item links, prices, every escalation step, and for searches with a known query, `query` (terms, matching items, text mentions) and `off_query`. `agreed` means the climb stopped because two browsers read the same page as the first read. - **`wall_vendor`** names the bot-defence service behind a wall the read met (`cloudflare`, `akamai`, `datadome`, `perimeterx`, `imperva`, `kasada`, `aws_waf`, `vercel`, `sucuri`), from the response's own headers and challenge page. `routing.learned_for` shows when that vendor's past results on this machine, or the bundled priors (`priors: true` when they changed the order), [reordered the ladder](/docs/blocked/escalation/#skipping-ahead), and `routing.escalated_to` names the tool that failed tries moved ahead (one that fetches from its own network) or walls moved ahead (your [real browser](/docs/blocked/real-browser/) when allowed, else a paid one). - **`rescued`** appears when the URL answered not found, or showed results that ignored its query, and the site's own search found what it asked for: `from`, `after` (`NOT_FOUND` or `off_query`), `query`, `via`, `search_url` ([When the URL is wrong](/docs/blocked/escalation/#when-the-url-is-wrong)). `rescue` (on the original receipt) says what was tried when nothing fit. - **`egress`** appears whenever a walled read tried [your other connections](/docs/blocked/egress/): `used` (the name of the one that got through, or `null`), `tried`, `after` and `direct_trace_id`, or `from_hint` when the site's hint chose it. - **`hedge`** appears when a read with no page after 3 seconds was raced by other free tools ([smart lightning](/docs/blocked/escalation/#racing-a-slow-read-smart-lightning)), or a [lightning](/docs/read/lightning/) read raced from the start (`lightning`): `providers`, `tried`, `after_seconds`, `provider`, `won`, and when its page stood in for a failed main read, `after_failure` and the `walled` tools. `routing.left_to_racers` lists the tools the main read left to its racers rather than read the page twice. - **`completeness.cut_off`** is true when a tool trying to complete a page ran past `completeness_step_timeout_seconds` (or the climb's deadline) and was stopped; the best page so far stands. - Cookies, keys and credential URL parameters never appear. ## Quality Every observed page read carries `receipt.quality`: a grade your agent can route on. ```json { "grade": "poor", "kind": "article", "flags": ["paywall"], "reason": "paywalled: only 433 characters of the article are free", "article": { "chars": 433, "paragraphs": 2, "method": "trafilatura", "paywall": true } } ``` - `good`: the content is there. - `partial`: it is there, with a warning (`needs_interaction`, `archived`). - `poor`: a placeholder (`$0` prices, template values), menus only, a cookie notice, results that ignore the query (`off_query`), or a paywall teaser. Article pages (schema.org `Article`/`NewsArticle`, Open Graph `article`, or publisher-style paths) are judged on their extracted article: 600+ characters and two or more prose paragraphs, however long the menus around them. A thin article escalates to a stronger tool first. On 108 fresh news and company articles the grade matched whether the article was there on 107. ## Freshness | `freshness` | What happens | | --- | --- | | `now` (default) | Always fetches. | | `hour`, `day`, `cached` | May reuse a saved copy. The receipt keeps the original time and reports its age. | An old copy is never passed off as new. ## Unknown stays unknown A page loading doesn't prove what's on it is current, and a 404 doesn't prove why it's gone. `field_status` keeps availability and price `unknown`; the caller decides what the page means. ## Images With `include_images=True`, images are downloaded and checked, not just linked. Each one has a hash, its real format and size, and its own error if it failed. `max_images` (default 50) caps the count; it does not mean the gallery is complete. ## Traces Every read saves a trace. `frankensurf trace ` or `Runtime.trace(trace_id)` shows one. `Runtime.capabilities()` adds traces up into success and speed per site and provider, with sample counts. Evidence lives under the state directory (`--state`, default `state`; the MCP server uses `FRANKENSURF_STATE`). It is never sent anywhere. --- # Settings Every option a read, search or action accepts, with its default. Every option is a field on `WebPolicy`, with a default. Nothing is hard-coded: if a read stops, waits or skips something, an option here decides it. - **Python**: pass `policy_overrides={...}` with only the options you mean. - **CLI**: common options have flags; see [Command line](/docs/reference/cli/). - **MCP**: pass them in `acquisition_policy`. ```python await web.read(url, policy_overrides={"allow_paid_fallbacks": True, "max_cost_usd": 0.05}) ``` Passing a whole `WebPolicy` instead fixes every option, which overrides a site's route seed and returns `EXPLICIT_POLICY_CONFLICT` where they disagree. These tables are generated from the code with `scripts/web_policy_doc.py`. ## Fetching | Field | Default | What it does | | --- | --- | --- | | `freshness` | `"now"` | `now` always fetches; `hour`, `day` and `cached` may reuse a saved copy. | | `render` | `False` | Require a browser; skips providers that cannot render. | | `prefer_markdown` | `False` | Ask servers for `text/markdown` first (T0). | | `sign_requests` | `True` | Sign plain HTTP reads with Web Bot Auth when a key exists (T3). | | `timeout_seconds` | `25` | Time allowed for each provider attempt. | | `unblocker_timeout_seconds` | `90.0` | Time allowed for an unblocker attempt. | | `agent_provider_timeout_seconds` | `300.0` | Time allowed for an agent provider such as Skyvern. | | `max_bytes` | 40 MB | Largest response accepted. | | `pdf_max_pages` | `50` | Most PDF pages to extract text from. Also the most pages a paid tool such as Firecrawl is asked to parse, since it bills per page. A longer PDF is cut short and says so: `structured.truncated` and `receipt.pdf`. | | `capture_json_responses` | `False` | On rendered reads, return the JSON the page fetched for itself as `captured_json`. | | `capture_json_max_items` | `20` | Most captured JSON responses kept. | | `capture_json_max_bytes` | 2 MB | Largest captured JSON response kept. | | `include_images` | `False` | Download and check the page's images. | | `max_images` | `50` | Most images to download. | | `card_images` | `False` | Add `cards`: each result link with its title and own thumbnail URL. | | `include_links` | `False` | Add `links`: every link on the page (`url`, `text`), in page order, once each. | | `max_links` | `500` | Most links `include_links` returns. | | `accept_language` | `None` | Accept-Language for plain HTTP reads, and the browser's locale (`de-DE,de;q=0.9`). | | `browser_actions` | `()` | Actions in the local browser before capture: wait, click, write, press, scroll, screenshot, scrape, executeJavascript ([Firecrawl-compatible API](/docs/firecrawl-compatible/#scrape-options)). Public reads only; use with `provider="local"`. | | `screenshot_full_page` | `False` | The browser screenshot covers the whole page. | | `markdown_output` | `False` | Add `markdown`: the page converted to markdown (headings, lists, links, tables). | | `scroll_screens` | `0` | Screens a browser scrolls before capture so lazy content loads (3 with `card_images`). | | `expect_terms` | `()` | Words search results should mention besides the URL query; results that mention none are flagged `off_query`. | | `main_content` | `False` | Also return `main_text`: the article without menus, footers and banners. | | `interactions` | `()` | Safe browser actions on browser reads: `dismiss_consent` (reject, never accept), `load_more`, `reveal`. | | `auto_items` | `False` | Items from any listing page with no saved module, found in the page's own data; the draft module comes back too. | | `max_image_bytes` | 20 MB | Largest image accepted. | | `image_max_attempts` | `2` | Attempts per image. | | `image_retry_delay_seconds` | `0.25` | Delay between image attempts. | | `image_retry_failures` | `PROVIDER_DOWN`, `TIMEOUT` | Image failures worth another attempt. | | `retain_public_failure_evidence` | `True` | Keep evidence of failed public reads. | ## Routing and cost | Field | Default | What it does | | --- | --- | --- | | `provider` | `None` | Force one provider. Never falls back. | | `provider_candidates` | `None` | An exact ordered list of providers. | | `allow_local_browser` | `True` | Allow providers that run a browser on your machine. | | `allow_paid_fallbacks` | `False` | Allow paid providers to join the route. | | `max_cost_usd` | `None` | Cap on measured provider cost for the whole operation. | | `strict_cost_cap` | `False` | A paid call that bills more than `max_cost_usd` still returns its valid content, and `receipt.budget` says the cap was exceeded. `True` discards it and fails with `BUDGET_EXHAUSTED`. | | `allow_handoff` | `False` | Try `handoff` after every automatic provider fails. | | `handoff_timeout_seconds` | `300.0` | How long a handoff waits for a person. | | `allow_real_browser` | `False` | Let your real Chrome or Edge join automatic reads; it moves ahead once walls do. | | `real_browser_wait_seconds` | `15.0` | How long the real browser lets a challenge clear by itself (it never clicks). | | `use_egress_pool` | `True` | Let a walled read try your other connections (FRANKENSURF_EGRESS). | | `egress_failures` | `BLOCKED`, `CAPTCHA`, `RATE_LIMITED` | Failures that start a pass through your other connections. | | `egress_tools` | `camoufox`, `scrapling`, `http` | Tools a pass through another connection uses; the wall vendor's best goes first. | | `allow_archive` | `False` | A stored copy (Internet Archive) as the very last resort; the result says when it was taken. | | `block_private_network` | `False` | Refuse loopback, private and link-local addresses, names that resolve to them, and redirects into them. | | `exclude_providers` | `()` | Skip these providers (the agent's try-harder lever). | | `try_harder` | `False` | Order the remaining providers strongest first, paid ones included when allowed. | | `retry_of_trace` | `None` | The trace this read retries, for the receipt. | | `terminal_failures` | `AUTH_REQUIRED`, `AUTH_EXPIRED`, `NOT_FOUND` | Failures that stop the climb. | | `context_stop_failures` | `AUTH_REQUIRED`, `AUTH_EXPIRED` | Failures that stop later reads of the same domain in this Runtime. | | `auth_wall_confirmations` | `5` | Providers in a row that must hit a sign-in wall before a public read stops. | | `fake_wall_confirmers` | `jina_reader`, `camoufox`, `firecrawl`, `zyte`, `scrapling`, `zenrows` | Providers that go next after a suspected fake wall, in this order. | | `escalate_after_walls` | `4` | Walls in one read before paid providers move ahead, and failed tries of any kind before free tools that fetch from their own network do; 0 turns both off. | | `paid_after_walls` | `1` | Walls before allowed paid tools move ahead, cheapest likely success first (one more when a free tool usually beats that wall vendor); 0 leaves them to escalate_after_walls. | | `escalation_failures` | `BLOCKED`, `CAPTCHA` | Failures that count as walls. | | `use_route_memory` | `True` | Let route memory reorder providers. | | `use_wall_priors` | `True` | Start wall-vendor routing from measured priors on a fresh install; your own reads outweigh them. | | `route_memory_ttl_seconds` | `3600` | How far back route memory looks. | | `route_memory_min_samples` | `3` | Clean reads in a row before a provider is promoted. | | `origin_route_hint_ttl_seconds` | `86400.0` | How long a site hint lasts. | | `provider_max_attempts_per_candidate` | `2` | Attempts per provider. Paid and capped reads are never retried. | | `provider_retry_delay_seconds` | `1.0` | Delay between attempts. | | `provider_retry_failures` | `PROVIDER_DOWN` | Failures worth another attempt. | | `provider_deadline_grace_seconds` | `10` | Extra time an isolated worker gets past `timeout_seconds`. | | `provider_cleanup_grace_seconds` | `5` | Time allowed to close a provider's process tree. | | `provider_composition_max_depth` | `8` | Deepest nesting when one provider calls another. | | `provider_composition_max_attempts` | `64` | Most nested provider attempts in one operation. | | `compound_source_candidates` | `http`, `camoufox`, `scrapling` | Validated provider list reserved for compound routes; not used by the default route. | ## Completeness and walls | Field | Default | What it does | | --- | --- | --- | | `completeness_escalation` | `True` | Check each automatic read's structure and re-read an incomplete page along the ladder. | | `completeness_ladder` | `jina_reader`, `scrapling`, `camoufox`, `firecrawl`, `zyte`, `zenrows`, `scrapfly`, `patchright`, `brightdata_unlocker` | Providers an incomplete page is re-read with, in order (paid ones only when allowed). | | `completeness_max_extra_reads` | `5` | Most extra reads one escalation makes. | | `completeness_parallel` | `2` | Free ladder tools raced at once (up to 4); paid tools always run alone. | | `completeness_stop_on_agreement` | `True` | Stop climbing once two browsers read the same page as the first read. | | `completeness_borderline_items` | `20` | A search page that passes with fewer item links (and few prices) gets one more read. | | `completeness_settle_ms` | `5000` | Minimum settle time for escalation reads. | | `completeness_deadline_seconds` | `150.0` | Time allowed for the whole escalation; a tool still running then is cut off. | | `completeness_step_timeout_seconds` | `45.0` | Once a read holds a page, time each extra tool gets to complete it before it is cut off. | | `hedge_after_seconds` | `3.0` | After this long without a page, a read is raced by hedge_racers free tools (smart lightning); 0 turns it off. | | `hedge_racers` | `2` | Free tools that race a read with no page after hedge_after_seconds (smart lightning), 1 to 3. | | `lightning` | `False` | Start the read and two other free tools at once and keep the first complete page; escalation races four. | | `rescue_not_found` | `True` | After NOT_FOUND, search the site itself for what the URL asked for and return the results when they mention it (receipt.rescued). | | `interact_on_escalation` | `True` | Pages that need a choice or show only a consent notice get one local-browser read with the safe actions first. | | `warm_up_on_wall` | `True` | A public read that ends at a wall gets one more try through the site's home page. | | `site_session_hours` | `24.0` | Hours the local browser keeps each site's cookies; 0 turns it off. | ## Pacing | Field | Default | What it does | | --- | --- | --- | | `origin_min_interval_seconds` | `2.0` | Minimum gap between reads of one origin. | | `origin_cooldown_seconds` | `900.0` | Pause after a wall. | | `origin_cooldown_failures` | `BLOCKED`, `CAPTCHA`, `RATE_LIMITED` | Failures that start a cool-down. | ## Rendering and content checks | Field | Default | What it does | | --- | --- | --- | | `wait_selector` | `None` | Wait for this CSS selector before capture. | | `wait_state` | `"attached"` | `attached` or `visible`, for `wait_selector`. | | `content_ready_selector` | `None` | Preferred ready selector; on timeout the current page is captured. | | `content_ready_timeout_seconds` | `10` | How long to wait for `content_ready_selector`. | | `settle_ms` | `400` | Extra wait after the page is ready. | | `public_browser_headless` | `True` | Run public browsers headless. | | `html_shell_min_text_chars` | `200` | A plain-HTTP page with scripts and less text than this is treated as an app shell. | | `html_shell_large_bytes` | `250000` | A page at least this large ... | | `html_shell_large_min_text_chars` | `1000` | ... with less text than this is also an app shell. | | `rendered_min_text_chars` | `50` | A rendered page with less text than this is `EMPTY_PAGE`. | | `second_opinion_text_chars` | `5000` | Below this much text, a plain HTTP page with scripts is also rendered and the fuller page kept; 0 turns it off. | | `navigation_page` | `1` | Page 1 to 3 through a site's own Next navigation (seeded routes only). | | `public_entry_url` | `None` | Entry page a `camoufox_entry` route visits first. | | `public_entry_continue_failures` | `BLOCKED` | Entry-page failures that still continue to the target. | | `scrapling_navigation_wait_until` | `None` | Navigation wait condition for Scrapling and entry-page routes. | | `scrapling_solve_cloudflare` | `False` | Let Scrapling attempt Cloudflare's interstitial. | | `scrapling_load_dom` | `True` | Wait for Scrapling's DOM load. | | `scrapling_google_search` | `False` | Scrapling's Google referrer option. | ## Search and paging | Field | Default | What it does | | --- | --- | --- | | `search_source_candidates` | `None` | An exact ordered list of search sources. | | `search_source_allow` | `None` | Only these sources may run. | | `search_source_prefer` | `()` | Move these sources to the front. | | `search_max_attempts` | `None` | Most sources to try. | | `search_source_timeout_seconds` | `8` | Time per source; `None` uses the operation timeout. | | `search_terminal_failures` | `POLICY_DENIED`, `BUDGET_EXHAUSTED` | Failures that stop search fallback. | | `search_merge_sources` | `4` | Sources asked at once by `search(mode="merge")`. | | `search_ladder` | `True` | An automatic search climbs sources until a result set passes the checks (on the query, inside the date window, real pages, deduplicated), carrying passing results up; false keeps the first answer. | | `search_good_results` | `3` | Passing results that make a set good (or the limit, if lower). | | `search_date_reads` | `6` | A windowed automatic search (`recency`) reads up to this many undated results' pages per source's result set (twice as many per search) over plain http, with no paid tool, to find their publication date; results provably outside the window drop out, and ones still undated count for at most half of `search_good_results`. 0 turns it off. | | `search_date_read_timeout_seconds` | `5.0` | Time one such date read gets, in seconds. | | `search_hold_undated` | `True` | In a windowed automatic search, results whose publication date could not be verified are left out of `results` whenever any verified in-window results exist, and returned apart as `unverified_results`. False mixes them in, ranked after the verified ones. | | `search_fill_rungs` | `2` | In a windowed automatic search, once the answer is good, up to this many more free sources are asked while it holds fewer verified in-window results than the limit, and what passes is merged in. Without a window, one more free source is asked when local SearXNG is reachable and the answer holds fewer passing results than the limit. Never a paid or last-resort source. 0 turns it off. | | `search_hedge_after_seconds` | `3.0` | After this long without an answer, the next free search source starts beside the current one; 0 turns it off. | | `search_hedge_racers` | `2` | Search sources running at once while racing, 1 to 3. Paid sources never race. | | `search_max_seconds` | `30.0` | Ceiling for one whole search, in seconds. | | `search_max_sources` | `6` | Ceiling on search sources started in one search. | | `search_paid_after_weak` | `2` | Weak or empty free sources before allowed paid search sources join, cheapest likely success first; 0 leaves them until every free source has been tried. | | `search_paid_parallel` | `True` | With `allow_paid_fallbacks`, an automatic search starts the cheapest likely-success paid search source beside the first free rung instead of after the free ones fail, whatever the kind of query. Everything is merged, checked and ranked, and a paid result that fails the checks gives way to a passing free one. The cost cap applies. False keeps paid sources behind the free rungs. | | `search_paid_wait_seconds` | `10.0` | When the free sources make the answer good while the parallel paid source is still running, wait this many seconds for it, so the answer can be anchored on it. If it does not answer in time, the free answer stands, `search_ladder.anchor` is null with the note `paid_timeout`, and the cancelled call is costed at its bundled price per search (marked estimated). 0 does not wait. | | `search_learned_routing` | `True` | Try first the source that most often passed the checks for this kind of query (vertical, recency, site filter). | | `search_exclude_operators` | `3` | Most -site: operators sent to an engine; the whole exclude list is enforced on the results either way. | | `max_pages` | `2` | Most pages `paginate` follows. | ## Browser agent | Field | Default | What it does | | --- | --- | --- | | `browser_agent_max_steps` | `40` | Most agent steps. | | `browser_agent_max_model_calls` | `40` | Most model calls. | | `browser_agent_max_actions` | `80` | Most actions overall. | | `browser_agent_max_actions_per_step` | `3` | Most actions per step. | | `browser_agent_max_failures` | `3` | Failures before the agent stops. | | `browser_agent_allowed_actions` | `navigate_public`, `inspect_page`, `follow_link`, `click_element`, `search_site`, `wait_readiness`, `scroll_page`, `done` | Actions the agent may take. | | `browser_agent_allowed_origins` | `None` | Origins the agent may visit; defaults to the target's. | | `browser_agent_entry_url` | `None` | Where the agent starts, if not the target URL. | | `browser_agent_task` | `None` | A task for the Browser Use agent. | | `browser_agent_llm_timeout_seconds` | `20` | Time per model call. | | `browser_agent_step_timeout_seconds` | `30` | Time per step. | | `browser_agent_action_timeout_seconds` | `10` | Time per action. | | `browser_agent_readiness_poll_ms` | `100` | Readiness polling interval. | | `browser_agent_packet_max_bytes` | 48 MB | Largest agent result packet. | | `browser_agent_use_vision` | `False` | Send screenshots to the model. | ## Identity and actions | Field | Default | What it does | | --- | --- | --- | | `identity` | `None` | Read as a registered identity. | | `profile` | `None` | Read as a stored FrankenSurf profile; exclusive with `identity`. | | `action_classes` | `READ_PUBLIC`, `READ_AUTHENTICATED` | Action classes this operation grants. | | `browser_do_allowed_tools` | `fill`, `click`, `wait_for`, `assert_text`, `assert_value` | Browser tools `do` may use. | | `browser_do_allowed_contracts` | `local_fixture.reversible_draft.v1` | Action contracts `do` may run. | | `browser_do_allowed_origins` | `None` | Exact origins `do` may act on. | | `browser_do_max_actions` | `20` | Most actions in one intent. | | `browser_do_action_timeout_seconds` | `10` | Time per action. | | `browser_do_settle_ms` | `200` | Wait after each action. | | `browser_do_packet_max_bytes` | 48 MB | Largest action intent. | | `browser_do_journal_max_bytes` | 16 MB | Largest action journal. | | `browser_do_selector_max_bytes` | `4096` | Longest selector. | | `browser_do_value_max_bytes` | 1 MB | Largest field value. | | `browser_do_description_max_bytes` | `8192` | Longest action description. | | `browser_do_idempotency_key_max_bytes` | `256` | Longest idempotency key. | | `browser_do_require_auth_check` | `True` | Require the identity's signed-in check before acting. | ## Repair | Field | Default | What it does | | --- | --- | --- | | `workload_assertion_max_count` | `32` | Most assertions on one extract (for repair). | | `workload_assertion_max_bytes` | `65536` | Largest assertion set. | | `repair_overlay_registry_max_bytes` | 4 MB | Largest repair overlay registry. | --- # Search Search the web with attributed results and automatic fallback across sources. `search` returns ranked results with their URL, title and snippet. Every result says which source found it, and every result is checked: inside the date window you asked for, about what you asked for, and an actual item rather than a homepage, listing or challenge page. - Free by default: your own [SearXNG](/docs/integrations/searxng/) (optional, no Docker: `frankensurf install search`, and search is stronger with it), then [DuckDuckGo](/docs/integrations/duckduckgo/) and [Bing](/docs/integrations/bing/), plus [Google News](/docs/integrations/google-news/), Wikipedia, Hacker News, Stack Exchange, GitHub, arXiv, Open Library, and `site_watch`, which reads a site's own feeds and sitemaps. - When a source's results fail the checks, Frankensurf climbs to the next one and merges what passed ([the search ladder](#the-search-ladder)). - Paid sources [Exa](/docs/integrations/exa/), Firecrawl search, [Brave](/docs/integrations/brave/), [Tavily](/docs/integrations/tavily/), [Parallel](/docs/integrations/parallel/), [Serper](/docs/integrations/serper/), [SerpAPI](/docs/integrations/serpapi/) and [Google Programmable Search](/docs/integrations/google-cse/) join when their keys are set and you allow paid tools, cheapest likely success first. - Name a source and Frankensurf uses only that one, never a substitute. ## Usage ```python results = await web.search("playwright python tutorial", limit=10) for hit in results["results"]: print(hit["title"], hit["url"]) ``` ```bash frankensurf search playwright python tutorial --limit 10 ``` ```json { "tool": "search", "arguments": { "query": "playwright python tutorial", "limit": 10 } } ``` ```json title="Response (trimmed)" { "query": "playwright python tutorial", "source": "searxng", "coverage": "partial", "results": [ { "url": "https://www.browserstack.com/guide/playwright-python-tutorial", "title": "The ultimate Playwright Python tutorial | BrowserStack", "snippet": "Learn end-to-end web testing with Playwright Python…", "engines": ["duckduckgo"], "verification": "indexed_discovery" }, { "url": "https://playwright.dev/python/docs/library", "title": "Getting started - Library | Playwright Python", "snippet": "These commands download the Playwright package and install browser binaries…", "engines": ["duckduckgo"], "verification": "indexed_discovery" } ], "upstream_failures": [["brave", "too many requests"], ["google cse", "too many requests"]] } ``` `coverage: "partial"` and `upstream_failures` show that some of SearXNG's own engines were rate-limited. Failures stay visible instead of looking like "no results". ## Narrow it down ```python results = await web.search("film camera repair", recency="week", region="en-AU", exclude_domains=["pinterest.com", "youtube.com"]) ``` ```bash frankensurf search film camera repair --recency week --region en-AU \ --exclude-domain pinterest.com --exclude-domain youtube.com ``` - `site`: only results from one domain (its subdomains too). - `exclude_domains`: up to 200 domains that never come back. Subdomains match too (excluding `example.com` drops `news.example.com`). Engines get only the first few as `-site:` operators (`search_exclude_operators`, default 3); the full list is enforced on the results, and the search asks for extra results so the limit is still met. - `recency`: `day`, `week`, `month` or `year`. - `region`: language and country, like `en-AU` or `de`. Each source applies what it can natively: SearXNG `time_range`, DuckDuckGo `df` and `kl`, Bing's date filters and `cc`, Exa and Tavily domain and date filters, Brave `freshness`, and `site:`/`-site:` operators where an engine reads them. Then every result list is filtered again, so an excluded domain never comes back from any source, and dated results older than the window are dropped. `receipt.search_options.filtered` counts what went, including how many results had no date to check. ## Boolean queries Monitoring tools write queries like `"Rio Tinto" OR "Rio Tinto Group" (acquisition OR layoffs OR "data breach")`. Each source gets the query as written. The checks judge results by what the query names: a story that names Rio Tinto (or Rio Tinto Group) is on the query without also naming an event word, which only narrows what the engines return. Event words (acquisition, layoffs, lawsuit, funding, earnings ...) beside a name make it a news search. Free sources often mishandle long OR-lists, returning homepages or nothing. When a boolean query's answer is not good, Frankensurf searches once more for the main name alone (`"Rio Tinto"`), in the same vertical and window, and keeps whichever answer passed more results. `receipt.query_simplified` says what was asked, what was kept and what the first try cost: ```json "query_simplified": {"from": "\"Rio Tinto\" OR ...", "to": "\"Rio Tinto\"", "kept": "simpler", "why": "the query as written got a weak answer (2 results passed the checks)"} ``` A named source is never retried. On 12 companies with free sources only and no SearXNG, a monitoring-style query found every company, against 7 of 12 before. ## Kinds of search `vertical` picks the kind of source. Automatic fallback stays inside it. Leave it out and FrankenSurf infers it from the query, so `fastify github repo` gets the code treatment and `who wrote Hamlet` the reference one. An explicit `vertical` always wins. | `vertical` | Source | Notes | | --- | --- | --- | | `web` (the fallback) | SearXNG, DuckDuckGo, Bing, plus paid sources when allowed | | | `news` | Bing News RSS | Publisher in `extra.publisher`, dates on every item | | `reference` | Wikipedia API | In the `region` language (`de` searches de.wikipedia.org). Question words are dropped (`who wrote War and Peace` searches for *War and Peace*), the article titled for the topic comes first, disambiguation pages are dropped, and titles with a qualifier such as `(film)` or `(1956 film)` go last unless the query asks for that kind of thing | | `discussions` | Hacker News (Algolia API) | `extra.points`, `extra.comments`, `extra.discussion_url` | | `qa` | Stack Exchange API | Stack Overflow by default; any Stack Exchange site by name via `engine_config={"engines": "superuser"}` | | `code` | GitHub API | Repositories; `recency` means pushed recently. `extra.stars`. Platform words (*github*, *repo*, *library*, *package*, *crate*, *source code*) and filler are not sent as search terms. A project name (`fastify github repo`) searches repository names; a need (`python library for image processing`) searches names and descriptions and turns a language word into a `language:` filter. Results are sorted by stars | | `papers` | arXiv API | | | `books` | Open Library | | All of these are official, keyless public APIs. ### Inferred verticals When no `vertical` is given, the query's wording picks one. The rules are deterministic and use common words and sentence shape only, never lists of sites or companies. A query that matches nothing is `web`. Rules are checked in this order and the first match wins. | Inferred | Signals | Example | | --- | --- | --- | | `qa` | An error or how-to (*how to*, *how do I*, *error*, *exception*, *fix*, *fails with*) together with code-shaped tokens (`snake_case`, `call()`, file extensions, error codes) or a language name | `how to reverse a list in python` | | `papers` | *papers on*, *paper on*, *research on*, *arxiv*, *preprint*, *survey of* | `papers on retrieval augmented generation` | | `code` | Platform words (*github*, *gitlab*, *repo*, *repository*, *crate*, *npm*, *pypi*, *source code*); *library for*, *framework for*, *sdk for*; a language beside *library*, *package*, *crate*, *module* or *framework* | `python library for image processing` | | `web` | Announcement phrasing (*announces*, *announcement*, *press release*, *launch*) stays windowed web, not news | `Anthropic announces new model` | | `news` | *news*, *headlines*, *latest on*; an entity with *today* or *this week* | `openai news` | | `reference` | Fact questions: *who*, *what is*, *what year*, *when was*, *how tall*, *how many*, *capital of*, *population of*, *currency of*, *wrote*, *invented*, *founded*, *painted*, *composed*, *discovered*; not when the query asks for *best*, *vs*, *review* or buying advice | `who wrote Hamlet` | A how-to with no code in it (`how to boil an egg`) stays `web`, as do words that are only sometimes about software (`Python the snake`, `Go to market strategy`, `package delivery tracking`). An explicit `source` is never re-routed. The receipt says what was picked and why: `receipt.search_routing.vertical_inferred` is `{"vertical": "code", "reason": "platform word: github"}`. It is absent when you passed a `vertical` yourself. `mode="merge"` infers the same way. A vertical with its own specialist source (`code`, `reference`, `papers`, `news`) does not wait for it to fail. When the local SearXNG is reachable, the specialist and SearXNG start together as the first rung, and their answers are merged, checked and ranked as one. If one fails or is slow it is simply left out (the slower one gets a few seconds after the first answer). Turning off `search_hedge_after_seconds` turns this racing off too. ## Local search Search works with nothing installed. A local SearXNG makes it stronger: `frankensurf install search` installs and starts one, no Docker needed, and search starts it again after a reboot. `frankensurf search-engine start`, `stop` and `status` manage it. See [SearXNG](/docs/integrations/searxng/). Expect it to thin out at volume. SearXNG asks the public engines from your own address, and Google, Brave and DuckDuckGo start refusing (CAPTCHAs, blocks) after a few dozen searches in a few minutes; in one test, 39 of the first 100 searches met a refusal. Frankensurf's settings switch on more engines (Bing, Yahoo, Qwant, Mojeek beside them) and rest a refusing engine for 15 minutes, so some keep answering, but for steady volume (hundreds of searches an hour) add a paid source such as [Serper](/docs/integrations/serper/) ($1 per 1,000) and allow paid tools. When an automatic search ran without it because nothing answered on `127.0.0.1:8088`, the receipt says why results may be thinner: ```json "search_routing": { "local_search": { "status": "not_running", "note": "Local search (SearXNG) was not running, so this search used other sources and may be thinner. Start it with: frankensurf search-engine start", "start_with": "frankensurf search-engine start" } } ``` The field is absent when SearXNG answered, when you named a source, and when you pointed at your own search URL. ## The search ladder An automatic search (no `source`) does not stop at the first source that answers. It checks every result set the way a read checks a page, and climbs to the next source until a set is good: at least `min(3, limit)` results (`search_good_results`) that pass. A result fails when it: - is provably outside the `recency` window (its own date, a date the snippet leads with, "3 days ago", or a `/2026/10/12/` URL path; an unknown date is not a failure, it only ranks below dated in-window results); - shares no key term with the query (stopwords and words like *news*, *latest*, *this week* don't count); - is a homepage, search page, tag listing or challenge page, when the query asks for specific items (a path that ends in an index word such as `/newsroom/`, `/changelog/`, `/press-releases/` or `/c/announcements/22`, or an aggregator's `/news/`, is a listing; a path with a title slug is not); - is about another thing with the same name: when the query's entity is a single word (*Block*, *Unity*, *Arm*), a result whose domain is not the entity's and whose title has the word inside a longer proper-noun phrase, led by a capitalised *The* (*The Block*) or following another capitalised word (*Cinder Block*), is `wrong_entity`. Possessives (*Dorsey's Block*), corporate suffixes (*Block, Inc.*) and freshness words don't count as a longer name; - repeats another result's page; - for an announcement query in a window (*announce*, *announcement*, *press release*, *launch*), is not an article: it passes only with a verified in-window date or an article-like path (a date in the path; a `news`, `press`, `blog`, `stories`, `announcements`, `releases`, `articles`, `updates`, `insights` or `media` segment followed by a slug of three or more hyphenated words or an id; or any slug of four or more words). Product, pricing, careers, platform, sign-in, investor overview, filings, results and category pages, listings (`?l=`, `?page=`), profiles and homepages fail as `not_article`, so the ladder climbs instead of returning them; - for a code query, is a fork or an archived repository, is in a language the query does not name (when it names one and GitHub knows the repository's), or has too few stars: under 500 for a need (*library for X*, *X framework*), under 100 for a name lookup unless the repository carries the name. With a `recency` window, results that carry no date are not left to guess at: the pages of up to `search_date_reads` (default 6) undated results that would make the top are read at once over plain http (`search_date_read_timeout_seconds`, default 5; no paid tools, the normal per-origin pacing), and the publication date in the page's own metadata (`article:published_time`, JSON-LD `datePublished`, ``; never a modified date) is set as the result's `published` with `date_source: "page"`. Results now provably outside the window drop out; results that still have no date stay, ranked below verified ones, and count for at most half of `search_good_results`, so the ladder keeps climbing instead of settling for undated filler. `receipt.search_checks.date_reads` reports `tried`, `dated`, `dropped`, `latency_ms` and `cost_usd`. `search_date_reads: 0` turns this off. When any result's date is verified inside the window, results whose date could not be verified leave `results` and come back apart as `unverified_results`: the answer holds only what is known to be fresh, and nothing is hidden. `search_hold_undated: false` mixes them back in, ranked last. So that holding results back does not leave the answer short, a good windowed answer with fewer verified results than `limit` asks up to `search_fill_rungs` (default 2) more free sources and merges what passes. Weak sets are not wasted: passing results from earlier sources are carried up, merged and deduplicated, and the search ends as soon as the sources together have enough. Paid sources join only with `allow_paid_fallbacks`, after `search_paid_after_weak` free sources (default 2) came back weak, cheapest likely success first, except for the queries below. A free source with no answer after `search_hedge_after_seconds` (default 3) is raced by the next free one; the first set that passes wins. `bing_rss` is the last web source and never races: when its results share nothing with the query it counts as failed, not as an answer. `search_max_seconds` and `search_max_sources` cap the climb. ### Ranking Passing results are ordered within each date tier by a relevance score (`check.relevance`). From the strongest signal down: 1. the entity, or every key term, in the title; 2. the result being the thing itself: its domain, or its GitHub `owner/repo`, is the entity or project name (among repositories of that name, more stars first); 3. for news and announcement queries, the entity's own site, and a press or news path on it; 4. key terms only in the snippet. Pages that are not what a search is after count against a result, from their URL and title alone, unless the query asks for that kind of page: stock quotes, share prices and liveblogs; policy and legal pages; release notes and documentation (for news and announcement queries); security advisory listings and news aggregator pages (the same); and *awesome-* lists and paper digests (for paper queries). They rank below every ordinary passing result. The source's own order breaks ties. Code queries add their own ordering. A query that asks for the repository (*github*, *repo*) prefers repository pages (`owner/name`) to an organisation's page or a project's website; repositories named *demo*, *example(s)*, *sample(s)*, *template*, *starter*, *boilerplate*, *tutorial*, *benchmark(s)* or *awesome-\** go last unless the query asks for them. For a name lookup (*httpx python github*) the repository whose name is the project, owner first, leads, and repositories whose owner/name does not hold the name as a word rank below those that do. For a need, popular repositories and pages that name libraries lead, and a how-to guide follows them. For a fact question in the reference vertical, a Wikipedia article whose title is not the question's topic (the topic of *who founded X* or *population of X* is X) ranks below every passing web result, so it fills a slot only when nothing else is left. #### Source priors with paid search allowed When paid sources are allowed, each result also carries the quality prior of the source it came from (`check.prior`), weighed with relevance in the merge. A paid source's prior is its bundled measured rate (`paid` in `bundled_wall_priors.json`); free sources use defaults (SearXNG 0.75, arXiv 0.8, Stack Exchange 0.7, Bing News and Wikipedia 0.6, GitHub 0.5, others 0.5), with Wikipedia 0.9 when the title is the topic and GitHub 0.85 for the project asked for or a repository with 1000 stars. A passing result of the best source therefore outranks a free one of similar relevance; a canonical free result, or one with a verified date against an undated paid one, can still lead. Results that fail the checks never count, whoever returned them. Without paid sources the ordering is unchanged. #### The answer starts from the paid source When a paid source answered, the answer starts from its results, in its own order. Each stays in place unless our checks fail it: off the query, outside the window, the wrong entity, a home page, listing or non-article, a stock blurb, a challenge page, or, for a code query, an organisation page or a project website when the repository was asked for, or a demo, example, template or similar repository that was not. A result that only ranks lower stays. The slots a failure leaves are filled, in order, by the best passing results of the other sources (a short paid set is filled the same way). For a name lookup or a reference question, a free canonical result the paid set lacks (the project's own repository, the article titled for the topic) takes a removed slot, or the last one. If no paid source answered, or paid is not allowed, the merge above applies unchanged. `receipt.search_ladder.anchor` records the source, how many results were kept, which were replaced and why, and how many slots the other sources filled. ### Paid search in parallel With `allow_paid_fallbacks`, an automatic search starts the cheapest likely-success paid source beside the first free rung, instead of after the free ones fail, whatever the kind of query. Everything is merged, checked and ranked, and a paid result that fails the checks gives way to a passing free one. `max_cost_usd` applies as always: a paid source that goes over the cap is dropped and the free results stand. `search_paid_parallel: false` keeps paid sources behind the free rungs. `receipt.search_ladder.paid_parallel` names the source that was started early. If the free sources make the answer good while that paid source is still running, the search waits for it, up to `search_paid_wait_seconds` (default 10), and anchors the answer on it. If it does not answer in time, the free answer stands, `receipt.search_ladder.anchor` is null with `notes: ["paid_timeout"]`, and the cancelled call is costed at its bundled price per search, marked `cost_basis: "estimated_bundled_price"` on its attempt and listed in `search_ladder.cost_estimated`. A source with no bundled price stays unknown. ### Filling a short answer Without a window, when the local SearXNG is reachable and a good answer holds fewer passing results than `limit`, one more free source (SearXNG first, if it has not been asked) is asked and merged in. With a window, up to `search_fill_rungs` more are, as above. Paid and last-resort sources are never asked to fill; `search_fill_rungs: 0` turns filling off. FrankenSurf remembers, per kind of query (vertical, recency or not, site filter or not), which source passed the checks and tries it first next time (`search_learned_routing`, stored in `search-stats.json` in the state directory). Every result carries `check` (`passed`, `reasons`, `in_window`, `on_query`, `published`, `date_source`, `kind`, `relevance`, and `prior` when paid sources are allowed), and the receipt says what happened: ```json { "search_ladder": {"enabled": true, "query_type": "news|recent|open", "order": ["bing_news_rss", "searxng", "duckduckgo_html"], "first_rung": ["bing_news_rss", "searxng"], "winner": "merged", "stopped": "good", "hedged": ["bing_news_rss", "searxng"]}, "search_checks": {"verdict": "good", "need": 3, "passed": 3, "reasons": {"off_query": 2, "out_of_window": 1}, "unknown_dates": 1, "results": [{"url": "https://example.com/2026/10/11/story", "source": "bing_news_rss", "in_window": true, "on_query": true, "published": "2026-10-11T23:59:59Z", "date_source": "url", "kind": "item"}]}, "search_attempts": [{"source": "bing_news_rss", "status": "observed", "count": 10, "passed": 2, "check": "weak", "cost_usd": 0, "latency_ms": 412}] } ``` `search_ladder: false` restores the first-answer behaviour. An explicit `source` is asked once and its results are annotated but never dropped. ## Merge mode `mode="merge"` asks every eligible source in the vertical at once (up to `search_merge_sources`, default 4) and fuses their lists by reciprocal rank. Duplicate pages (`www.`, trailing slashes and tracking parameters aside) merge into one result that lists every engine that found it, with `fusion.sources` giving each engine's rank. One blocked engine costs nothing, and a page found by several engines rises. `receipt.search_merge` reports each source. ```python results = await web.search("film camera repair sydney", mode="merge") ``` ## Options | Option | Default | Description | | --- | --- | --- | | `source` | automatic | One source only, by name (see the tables above). | | `limit` | `10` | Number of results. | | `site`, `exclude_domains`, `recency`, `region` | none | See above. | | `vertical` | inferred | The kind of source. Inferred from the query when left out; an explicit value always wins. | | `mode` | `fallback` | `merge` to ask every eligible source at once. | | `search_source_prefer` | none | Try these sources first. | | `search_source_allow` | all | Only these sources may run. | | `search_source_timeout_seconds` | `8` | Time per source. | | `allow_paid_fallbacks` | `false` | Let the paid sources join. | A search hit is an indexed candidate, not proof the page is live. Read it to find out. --- # Watch Poll a page and get only the links you haven't seen before. `watch` reads a listing, feed or search page and returns the same-site links that are new since your last poll. You decide when to poll and what a link means. - Filter links with a regular expression. - The first poll returns every current link and sets `first_poll`, so you can treat it as a baseline. - Each poll is one ordinary read: paced, receipted, and able to climb past walls. ## Many sites: `watch_sites` To keep up with many sites at once, `watch_sites` uses what each site publishes about its own new pages instead of reading its front page: - the feeds it advertises (``: RSS, Atom, JSON Feed); - the sitemaps `robots.txt` names (news sitemaps first, then the most recently changed); - a few conventional feed paths (`/feed`, `/rss.xml`, …) when it advertises none. They are found once per site and remembered for a week. Every poll fetches them with conditional requests (`If-None-Match`, `If-Modified-Since`), so an unchanged feed costs one small 304. Pages already reported are remembered, and the same story under two URLs (syndication) is reported once. ```python new = await web.watch_sites(["https://news.example.com/", "https://blog.example.org/"], since="2026-10-07", read_new=True) for page in new["pages"]: print(page["published"], page["url"], page["quality"]) ``` ```bash frankensurf watch-sites https://news.example.com/ https://blog.example.org/ --since 2026-10-07 ``` `read_new` reads the new pages too, with main content, as one paced batch. `requests` and `unchanged_feeds` in the answer say what a poll cost. ## One page: `watch` ## Usage ```python poll = await web.watch("https://news.ycombinator.com/", link_pattern=r"item\?id=\d+") for link in poll["new"]: print(link["url"]) ``` ```bash frankensurf watch https://news.ycombinator.com/ --link-pattern 'item\?id=\d+' ``` ```json title="Response (trimmed)" { "url": "https://news.ycombinator.com/", "status": "observed", "first_poll": true, "new": [ { "url": "https://news.ycombinator.com/item?id=49969183", "text": "4 hours ago" }, { "url": "https://news.ycombinator.com/item?id=49971230", "text": "1 hour ago" } ], "links_on_page": 30, "seen_count": 30, "receipt": { "status": "observed", "method": "http" } } ``` ## Options | Option | Default | Description | | --- | --- | --- | | `link_pattern` | all same-site links | Regular expression a link URL must match. | | `state_key` | the URL | Name of the watch, if you watch one URL several ways. | | `max_seen` | `5000` | Links remembered per watch. | Seen links are stored under `/watches/`. A few polls an hour is plenty for most pages. Pacing and cool-downs apply, so hammering a page only gets it paused.