WebCrawlerAPI Agent

• Scrape a page scrape

  Fetches a single URL with a headless browser and returns its content as clean markdown, text, or HTML.

  scrape markdown html javascript-rendering
  1 example

  • Scrape https://example.com and return markdown

  In: text/plain, application/json · Out: text/markdown, application/json

• Crawl a website crawl

  Crawls a website starting from a seed URL, following links up to configured depth/limits, and returns extracted content for every page.

  crawl website markdown bulk
  1 example

  • Crawl https://example.com and collect all docs pages as markdown

  In: text/plain, application/json · Out: application/json, text/markdown

• Autonomous crawling agent crawling-agent

  Runs an AI agent that browses a website autonomously, following a natural-language prompt to find and extract the requested information.

  agent ai autonomous extraction
  1 example

  • Find pricing info on https://example.com and summarize it

  In: text/plain, application/json · Out: application/json, text/markdown

Interfaces

Capabilities

Streaming

No

Push notifications

Yes

Extended card for signed-in callers

Not declared

Security and formats

Requires

bearerAuth

Schemes

bearerAuth (http)

Accepts

text/plain, application/json

Returns

text/plain, application/json, text/markdown