Type to search.

Firecrawl connector

Serverless Preview

Elastic Cloud Serverless projects are autoscaled environments, fully managed by Elastic and available on Elastic Cloud.

Availability

  • Preview

    This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply.

Serverless interfaces and procedures might differ from classic Elastic Stack deployments.

Stack Preview 9.4+

The Elastic Stack includes Elastic's core products such as Elasticsearch, Kibana, Logstash, and Beats.

Availability

  • Preview since 9.4

    This functionality is in technical preview and is ready for evaluation. Use with caution in production; it is not recommended for mission-critical workloads. Elastic will work to fix any issues, but features in technical preview are not subject to the support SLA of official GA features. Specific Support terms apply.

Unless stated otherwise on the page, this functionality is available when your Elastic Stack is deployed on Elastic Cloud Hosted, Elastic Cloud Enterprise, Elastic Cloud on Kubernetes, and self-managed environments.

This documentation corresponds to the latest patch available for each minor version. If you're not using the latest patch, check the release notes for changes.

The Firecrawl connector uses the Firecrawl REST API v2 to scrape web pages, search the web, map site URLs, and crawl websites. It supports Bearer token (API key) authentication.

You can use this connector in Agent Builder and Workflows.

You can create connectors in Stack Management > Connectors.

Firecrawl connectors have the following configuration properties:

API Key (Bearer token)
Your Firecrawl API key for authentication. The connector sends it in the Authorization: Bearer <key> header to https://api.firecrawl.dev.

You can test connectors when you create or edit the connector in Kibana.

The Firecrawl connector has the following actions:

scrape

Scrape a single URL and extract content (for example, Markdown or HTML).

  • url (required): The URL to scrape.
  • formats (optional): Output formats: markdown, html, links, screenshot, extract. Defaults to Markdown.
  • onlyMainContent (optional): Return only main content, excluding navigation and footer elements. Defaults to true.
  • waitFor (optional): Delay in milliseconds before fetching (useful for JavaScript-rendered pages).
search

Search the web and optionally get full content from results.

  • query (required): Search query string.
  • limit (optional): Maximum number of results (1 to 100). Defaults to 5.
map

Map a website to discover indexed URLs.

  • url (required): Base URL to map.
  • search (optional): Search term to filter or order URLs by relevance.
  • limit (optional): Maximum URLs to return. Defaults to 5000.
  • includeSubdomains (optional): Include subdomains. Defaults to true.
  • ignoreQueryParameters (optional): Deduplicate by path only. Defaults to true.
crawl

Start an asynchronous crawl of a website. Returns a job ID. Use getCrawlStatus to poll for results.

  • url (required): Base URL to start crawling from.
  • limit (optional): Maximum pages to crawl. Defaults to 100.
  • maxDiscoveryDepth (optional): Maximum discovery depth.
  • allowExternalLinks (optional): Follow external links. Defaults to false.
getCrawlStatus

Get the status and results of a crawl job.

  • id (required): The crawl job ID (UUID) returned from the crawl action.

Use the Action configuration settings to customize connector networking, such as proxies, certificates, or TLS settings. You can set configurations that apply to all your connectors or use xpack.actions.customHostSettings to set per-host configurations.

To use the Firecrawl connector:

  1. Sign up at Firecrawl or log in to your account.
  2. Go to API keys in the Firecrawl dashboard.
  3. Create an API key or copy an existing one (for example, fc-...).
  4. Enter the API key in the API Key field when configuring the Firecrawl connector in Kibana.

Rate limits and billing depend on your Firecrawl plan. Refer to Firecrawl billing for details.