Cloudsteading
Submit

Existing projects to consider

These repositories cover part of this workflow. Check their scope and hosting requirements before choosing.

Workers Firecrawl’s original OpenAPI interface, captured from the pinned app running locally. The displayed request body is a schema example; route validation and optional API-key guards were tested, but browser scraping was not.
Existing project · Paid hosting baseline

workers-firecrawl

By g4brym

Requires fixes before production use · see runtime findings on the project page

Search, link mapping and single-page browser scraping/content extraction; full crawl-job orchestration, LLM extraction and Apify actor-platform parity are excluded.

workers-firecrawl has a documented low-volume paid Cloudflare path beginning with Workers Paid ($5 USD/account/month), with separately metered usage. The upstream setup requires Paid; browser hours and concurrency must remain bounded.

Explore workers-firecrawl →
Build proposal

This is a proposed build, with a conceptual architecture. Any linked existing projects have their own documented scope and infrastructure.

Browse existing projects and remaining proposals →
Queue #1
Waiting for votes

Firecrawl on Cloudflare

Crawler, markdown extraction, and crawl jobs rebuilt with Workers, Queues, Browser Rendering, D1, and R2.

replaces FirecrawlBrowser RenderingD1QueuesR2WorkersWorkers AI
Firecrawl logo
would replace
Firecrawl· web-scraping

Web crawling and LLM-ready extraction APIs.

How it would work

A URL goes in, an API-ready result comes out. These are the stages the request flows through.

  1. input01

    Submit URL or crawl seed

    Send a URL, crawl depth, output format, and optional extraction hints.

  2. render02

    Render when needed

    Use Browser Rendering for JavaScript-heavy pages and plain Workers fetch for simple pages.

  3. extract03

    Clean into useful output

    Turn pages into markdown, metadata, links, screenshots, and optional structured JSON.

  4. store04

    Persist crawl history

    Keep job state in D1 and larger artefacts in R2 so results can be inspected later.

  5. deliver05

    Return API-ready results

    Expose job status, retry failures, and return outputs through a small dashboard/API.

Concrete jobs it does

If you're paying Firecrawl for any of these, the proposed build would let you stop.

  • →Scrape one URL into clean, LLM-ready markdown
  • →Crawl a small site with depth + page-count limits
  • →Render JavaScript-heavy pages (browser only when needed)
  • →Extract structured fields with Workers AI
  • →Store raw HTML, screenshots, and JSON artefacts in R2
  • →Expose an API with key-based quotas and a tiny dashboard
  • →Replay or rerun a job from job state in D1

Config knobs

What you'll be able to tune from day one.

Crawl mode
single page / small site / scheduled recrawl
Rendering
auto-detect, browser only when needed
Output
markdown, links, screenshot, structured JSON
Storage
D1 for jobs, R2 for artefacts
Limits
depth, page count, timeout, domain allowlist
Auth
API keys with per-key quota
proposed architecture

The Cloudflare loadout, wired together.

Architecture

Overview inferred from the listed Cloudflare services. Routes and connections have not been verified.

Public interface
Surfaces1
client / caller
interface not yet reviewed
↓
App
Workers
entry
request handler
↓
what each tool does
Cloudflare feature
Browser Renderingtool 01
JavaScript page renderer

Render dynamic pages before extracting text, links, screenshots, and page state.

D1tool 02
Job state database

Store users, API keys, crawl jobs, page status, errors, limits, and dashboard history.

Queuestool 03
Crawl job pipeline

Break a site crawl into retryable page jobs so long-running work does not depend on one request.

R2tool 04
Crawl artefact storage

Keep raw HTML, screenshots, cleaned markdown, JSON output, and debug artefacts cheaply.

Workerstool 05
API gateway + crawl coordinator

Expose scrape/crawl endpoints, validate API keys, route work, and assemble responses.

Workers AItool 06
Extraction assistant

Classify pages and extract structured fields when deterministic parsing is not enough.

will build
  • +Single-page scrape API returning cleaned markdown and metadata.
  • +Queued crawl jobs for small sites, with progress state and retries.
  • +JavaScript rendering path for pages that need a browser.
  • +Stored crawl outputs, raw artefacts, and simple API-key based access.
  • +A small dashboard to inspect recent jobs, failures, usage, and outputs.
will not build
  • -Massive managed crawl infrastructure for very large or hostile sites.
  • -A perfect clone of every Firecrawl endpoint and extraction option.
  • -Enterprise compliance, SLAs, team governance, or managed support.
  • -Guaranteed bypassing of bot protection, paywalls, CAPTCHAs, or site restrictions.
  • -Every weird website edge case on day one.
readme

Firecrawl on Cloudflare

Firecrawl is valuable because it turns messy websites into clean, LLM-ready data without making the team own crawling infrastructure. This candidate asks whether the practical small-team version can be rebuilt with Cloudflare primitives while staying honest about scale, reliability, and edge cases.

What this is

Firecrawl is useful because it hides crawl orchestration, page rendering, extraction, retries, and storage behind a simple API.

How we would build it

Use Browser Rendering, D1, Queues, R2, Workers to cover the core workflow without adding rented infrastructure.

The honest limit

Excellent for small teams, internal agents, and controlled crawl workloads. It will not immediately match Firecrawl's managed reliability or every extraction edge case.

Discussion · 0

sign in to comment →
No comments yet — be the first.