Firecrawl-compatible API

A lot of agent projects read the web through Firecrawl, and most of them let you point at your own Firecrawl server. frankensurf serve is one: it answers Firecrawl’s scrape and search calls, and reads every page through Frankensurf instead.

frankensurf serve            # http://127.0.0.1:3002, Firecrawl's own port, this machine only

Then set the project’s Firecrawl URL to http://127.0.0.1:3002. The API key can be anything; it isn’t checked. Free tools only unless you start it with --paid, and --max-cost caps a request.

What you get back

• Scrape (POST /v1/scrape, /v2/scrape): the page as markdown (headings, lists, links, tables, converted from its HTML), plus html and links if you ask for them. metadata.frankensurf carries the trace ID, so frankensurf trace <id> shows every tool it tried. onlyMainContent is accepted and ignored: Frankensurf’s main-content cut is built for articles, and on search and listing pages it lost the results. Menus cost a few tokens; missing results cost the answer.

• Search (POST /v1/search, /v2/search): Frankensurf’s checked search results. With scrapeOptions.formats, each result is read too.

• A page no tool could read is never passed off as one. Scrape answers success: false with a plain reason, status 500 like Firecrawl’s own failures (408 when the request’s timeout runs out), so the SDKs raise it as an error:

  Could not read this page (CAPTCHA: Provider returned a challenge or sign-in page).
  Tried: http, steel, camoufox, local, scrapling_http, patchright, nodriver, crawl4ai.
  Don't fill in its contents from search snippets or memory.

  In search, that result keeps its URL, title and description, but has no markdown; the reason is in metadata.error. A research agent that uses the markdown simply doesn’t get a page that wasn’t there.

• Map (/v1/map, /v2/map) and crawl (/v1/crawl, /v2/crawl, then GET /v1/crawl/{id} until completed; DELETE cancels): the same engine as Map and crawl. Crawl jobs live in the server’s memory, so they end with it. Pages no tool could read are listed under frankensurf.failed.

• With a model (frankensurf serve --model <any LiteLLM model>, or FRANKENSURF_AGENT_MODEL): the summary and json scrape formats (v1 jsonOptions, v2 {"type": "json", "schema", "prompt"}), and /v1/extract, /v2/extract jobs (GET /v1/extract/{id}). See Agent mode.

• Batch scrape (/v1/batch/scrape, /v2/batch/scrape, then GET .../batch/scrape/{id}): each URL scraped with the same options, paced per site; failures under frankensurf.failed.

Scrape options

Firecrawl option	In Frankensurf
actions (wait, click, write, press, scroll, screenshot, scrape, executeJavascript)	Run in order in Frankensurf’s own local browser before the page is captured; results under actions (screenshots, scrapes, javascriptReturns). Public reads only: never with a stored profile or identity, where a click could act as you. A failed action stops the rest and is named in metadata.frankensurf.action_failed.
screenshot, screenshot@fullPage, {"type": "screenshot", "fullPage": true}	A browser read; the image comes back as a data:image/png;base64,... URL rather than a hosted link.
changeTracking (modes: ["git-diff"], tag)	Compared with the last scrape of the same URL and tag on this machine: new, same or changed, with previousScrapeAt and a unified diff. Kept in the state directory, so it survives restarts.
location.languages	Sent as Accept-Language and used as the browser’s locale. location.country isn’t applied: Frankensurf has no proxies of its own in other countries (egress can add some).
proxy: "stealth"	Starts from the stealth browsers (Scrapling, Camoufox, Patchright, nodriver) instead of plain HTTP. basic and auto are the usual cheapest-first route.
webhook (crawl, batch scrape)	started, page, completed and failed events, POSTed as {success, type, id, data, metadata}, filtered by events, with your headers.

Crawl, batch scrape and extract jobs are kept in the state directory: their results outlive the server, and a job that was running when it stopped comes back as failed, saying so.

GPT Researcher

GPT Researcher can scrape through Firecrawl. Point it at Frankensurf:

frankensurf serve &
export SCRAPER=firecrawl
export FIRECRAWL_SERVER_URL=http://127.0.0.1:3002
export FIRECRAWL_API_KEY=local
export FIRECRAWL_CONCURRENCY=4     # GPT Researcher's default of 2 suits Firecrawl's free tier
pip install firecrawl-py           # GPT Researcher installs it on first use otherwise

That changes the page reads. For search too, install the retriever plugin, which ships the way GPT Researcher asks search providers to ship:

pip install "gptr-frankensurf @ git+https://github.com/yail259/frankensurf#subdirectory=contrib/gptr-frankensurf"
export RETRIEVER=frankensurf          # or frankensurf,duckduckgo

Its results are Frankensurf’s checked search results, free sources only unless FRANKENSURF_GPTR_PAID=1.

What it changed, on 150 sites

We read the 150 sites of our held-out set 7 (picked before any was read, never tuned on) with GPT Researcher’s own scraper code, and graded every page with the benchmark’s check: does the text hold what the page should, at least three times.

GPT Researcher scraping with	Right page	Wrong page passed on	No page
its default scraper (SCRAPER=bs)	58	15	77
frankensurf serve, free tools only	134	3	13, each with the reason

Through Frankensurf it read 76 sites the default scraper didn’t, and none the other way. It is slower: a median of 9.0 s a page against 1.7 s, and 40 s for the slowest tenth, because a walled page climbs through several tools before it gets in or gives up.

We also had it write full reports for 8 questions whose best sources sit behind walls. Through frankensurf serve it read 89 sources against 65, and a model grading the reports blind scored them 7.0 against 5.9 out of 10, better on 6 questions and worse on 1. That’s a consistent direction, but 8 questions are too few to call it significant. One run on 16 October 2026; the rows and scripts are in benchmarks/2026-10-16-gpt-researcher.

deep-research

deep-research searches and reads through Firecrawl. In its .env.local:

FIRECRAWL_KEY=local
FIRECRAWL_BASE_URL=http://127.0.0.1:3002
FIRECRAWL_CONCURRENCY=4

Its searches ask for each result’s markdown with a 15 second timeout. Frankensurf answers 3 seconds inside it, so a slow page comes back as a missing page instead of a failed search. With its exact call, four searches got all 20 pages in time, twice, in 4 to 13 seconds a search. (Before v0.44.1 it was 17 and 14 of 20: news searches returned Google News redirect links, which are slow to follow. Search now swaps them for the publisher’s own URL.)

Anything else that takes a Firecrawl URL

We’ve checked Firecrawl’s own SDKs against it: @mendable/firecrawl-js 1.16 (v1 API) and firecrawl-py 4.50 (v2 API), for scrape and search. If a tool lets you set Firecrawl’s URL, it should work; tell us if one doesn’t.

Note

serve listens on 127.0.0.1 only. --host 0.0.0.0 opens it to your network, with no authentication: anyone who can reach it can read pages through your machine.