# Firecrawl-compatible API Run Frankensurf as a local Firecrawl-compatible endpoint, so GPT Researcher, deep-research and anything else that takes a Firecrawl URL reads through it with no code change. A lot of agent projects read the web through Firecrawl, and most of them let you point at your own Firecrawl server. `frankensurf serve` is one: it answers Firecrawl's `scrape` and `search` calls, and reads every page through Frankensurf instead. ```bash frankensurf serve # http://127.0.0.1:3002, Firecrawl's own port, this machine only ``` Then set the project's Firecrawl URL to `http://127.0.0.1:3002`. The API key can be anything; it isn't checked. Free tools only unless you start it with `--paid`, and `--max-cost` caps a request. ## What you get back - **Scrape** (`POST /v1/scrape`, `/v2/scrape`): the page as markdown (headings, lists, links, tables, converted from its HTML), plus `html` and `links` if you ask for them. `metadata.frankensurf` carries the trace ID, so `frankensurf trace ` shows every tool it tried. `onlyMainContent` is accepted and ignored: Frankensurf's [main-content cut](/docs/read/main-content/) is built for articles, and on search and listing pages it lost the results. Menus cost a few tokens; missing results cost the answer. - **Search** (`POST /v1/search`, `/v2/search`): Frankensurf's checked search results. With `scrapeOptions.formats`, each result is read too. - **A page no tool could read is never passed off as one.** Scrape answers `success: false` with a plain reason, status 500 like Firecrawl's own failures (408 when the request's `timeout` runs out), so the SDKs raise it as an error: ```text Could not read this page (CAPTCHA: Provider returned a challenge or sign-in page). Tried: http, steel, camoufox, local, scrapling_http, patchright, nodriver, crawl4ai. Don't fill in its contents from search snippets or memory. ``` In search, that result keeps its URL, title and description, but has no markdown; the reason is in `metadata.error`. A research agent that uses the markdown simply doesn't get a page that wasn't there. - **Map** (`/v1/map`, `/v2/map`) and **crawl** (`/v1/crawl`, `/v2/crawl`, then `GET /v1/crawl/{id}` until `completed`; `DELETE` cancels): the same engine as [Map and crawl](/docs/crawl/). Crawl jobs live in the server's memory, so they end with it. Pages no tool could read are listed under `frankensurf.failed`. - **With a model** (`frankensurf serve --model `, or `FRANKENSURF_AGENT_MODEL`): the `summary` and `json` scrape formats (v1 `jsonOptions`, v2 `{"type": "json", "schema", "prompt"}`), and `/v1/extract`, `/v2/extract` jobs (`GET /v1/extract/{id}`). See [Agent mode](/docs/agent-mode/). - **Batch scrape** (`/v1/batch/scrape`, `/v2/batch/scrape`, then `GET .../batch/scrape/{id}`): each URL scraped with the same options, paced per site; failures under `frankensurf.failed`. ### Scrape options | Firecrawl option | In Frankensurf | | --- | --- | | `actions` (`wait`, `click`, `write`, `press`, `scroll`, `screenshot`, `scrape`, `executeJavascript`) | Run in order in Frankensurf's own local browser before the page is captured; results under `actions` (`screenshots`, `scrapes`, `javascriptReturns`). Public reads only: never with a stored profile or identity, where a click could act as you. A failed action stops the rest and is named in `metadata.frankensurf.action_failed`. | | `screenshot`, `screenshot@fullPage`, `{"type": "screenshot", "fullPage": true}` | A browser read; the image comes back as a `data:image/png;base64,...` URL rather than a hosted link. | | `changeTracking` (`modes: ["git-diff"]`, `tag`) | Compared with the last scrape of the same URL and tag on this machine: `new`, `same` or `changed`, with `previousScrapeAt` and a unified diff. Kept in the state directory, so it survives restarts. | | `location.languages` | Sent as Accept-Language and used as the browser's locale. `location.country` isn't applied: Frankensurf has no proxies of its own in other countries ([egress](/docs/blocked/egress/) can add some). | | `proxy: "stealth"` | Starts from the stealth browsers (Scrapling, Camoufox, Patchright, nodriver) instead of plain HTTP. `basic` and `auto` are the usual cheapest-first route. | | `webhook` (crawl, batch scrape) | `started`, `page`, `completed` and `failed` events, POSTed as `{success, type, id, data, metadata}`, filtered by `events`, with your `headers`. | Crawl, batch scrape and extract jobs are kept in the state directory: their results outlive the server, and a job that was running when it stopped comes back as `failed`, saying so. ## GPT Researcher [GPT Researcher](https://github.com/assafelovic/gpt-researcher) can scrape through Firecrawl. Point it at Frankensurf: ```bash frankensurf serve & export SCRAPER=firecrawl export FIRECRAWL_SERVER_URL=http://127.0.0.1:3002 export FIRECRAWL_API_KEY=local export FIRECRAWL_CONCURRENCY=4 # GPT Researcher's default of 2 suits Firecrawl's free tier pip install firecrawl-py # GPT Researcher installs it on first use otherwise ``` That changes the page reads. For search too, install the retriever plugin, which ships the way GPT Researcher asks search providers to ship: ```bash pip install "gptr-frankensurf @ git+https://github.com/yail259/frankensurf#subdirectory=contrib/gptr-frankensurf" export RETRIEVER=frankensurf # or frankensurf,duckduckgo ``` Its results are Frankensurf's checked search results, free sources only unless `FRANKENSURF_GPTR_PAID=1`. ### What it changed, on 150 sites We read the 150 sites of our held-out set 7 (picked before any was read, never tuned on) with GPT Researcher's own scraper code, and graded every page with the benchmark's check: does the text hold what the page should, at least three times. | GPT Researcher scraping with | Right page | Wrong page passed on | No page | | --- | --- | --- | --- | | its default scraper (`SCRAPER=bs`) | 58 | 15 | 77 | | `frankensurf serve`, free tools only | 134 | 3 | 13, each with the reason | Through Frankensurf it read 76 sites the default scraper didn't, and none the other way. It is slower: a median of 9.0 s a page against 1.7 s, and 40 s for the slowest tenth, because a walled page climbs through several tools before it gets in or gives up. We also had it write full reports for 8 questions whose best sources sit behind walls. Through `frankensurf serve` it read 89 sources against 65, and a model grading the reports blind scored them 7.0 against 5.9 out of 10, better on 6 questions and worse on 1. That's a consistent direction, but 8 questions are too few to call it significant. One run on 16 October 2026; the rows and scripts are in [`benchmarks/2026-10-16-gpt-researcher`](https://github.com/yail259/frankensurf/tree/main/benchmarks/2026-10-16-gpt-researcher). ## deep-research [deep-research](https://github.com/dzhng/deep-research) searches and reads through Firecrawl. In its `.env.local`: ```bash FIRECRAWL_KEY=local FIRECRAWL_BASE_URL=http://127.0.0.1:3002 FIRECRAWL_CONCURRENCY=4 ``` Its searches ask for each result's markdown with a 15 second timeout. Frankensurf answers 3 seconds inside it, so a slow page comes back as a missing page instead of a failed search. With its exact call, four searches got all 20 pages in time, twice, in 4 to 13 seconds a search. (Before v0.44.1 it was 17 and 14 of 20: news searches returned Google News redirect links, which are slow to follow. Search now swaps them for the publisher's own URL.) ## Anything else that takes a Firecrawl URL We've checked Firecrawl's own SDKs against it: `@mendable/firecrawl-js` 1.16 (v1 API) and `firecrawl-py` 4.50 (v2 API), for scrape and search. If a tool lets you set Firecrawl's URL, it should work; tell us if one doesn't. `serve` listens on 127.0.0.1 only. `--host 0.0.0.0` opens it to your network, with no authentication: anyone who can reach it can read pages through your machine.