•  


GitHub - mendableai/firecrawl: ?? Turn entire websites into LLM-ready markdown
Skip to content

mendableai/firecrawl

Repository files navigation

?? Firecrawl

Crawl and convert any website into LLM-ready markdown. Build by Mendable.ai

This repository is currently in its early stages of development. We are in the process of merging custom modules into this mono repository. The primary objective is to enhance the accuracy of LLM responses by utilizing clean data. It is not ready for full self-host yet - we're working on it

What is Firecrawl?

Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. We crawl all accessible subpages and give you clean markdown for each. No sitemap required.

Pst. hey, you, join our stargazers :)

How to use it?

We provide an easy to use API with our hosted version. You can find the playground and documentation here . You can also self host the backend if you'd like.

To run locally, refer to guide here .

API Key

To use the API, you need to sign up on Firecrawl and get an API key.

Crawling

Used to crawl a URL and all accessible subpages. This submits a crawl job and returns a job ID to check the status of the crawl.

curl -X POST https://api.firecrawl.dev/v0/crawl \
    -H 
'
Content-Type: application/json
'
 \
    -H 
'
Authorization: Bearer YOUR_API_KEY
'
 \
    -d 
'
{

      "url": "https://mendable.ai"

    }
'

Returns a jobId

{ 
"jobId"
: 
"
1234-5678-9101
"
 }

Check Crawl Job

Used to check the status of a crawl job and get its result.

curl -X GET https://api.firecrawl.dev/v0/crawl/status/1234-5678-9101 \
  -H 
'
Content-Type: application/json
'
 \
  -H 
'
Authorization: Bearer YOUR_API_KEY
'
{
  
"status"
: 
"
completed
"
,
  
"current"
: 
22
,
  
"total"
: 
22
,
  
"data"
: [
    {
      
"content"
: 
"
Raw Content 
"
,
      
"markdown"
: 
"
# Markdown Content
"
,
      
"provider"
: 
"
web-scraper
"
,
      
"metadata"
: {
        
"title"
: 
"
Mendable | AI for CX and Sales
"
,
        
"description"
: 
"
AI for CX and Sales
"
,
        
"language"
: 
null
,
        
"sourceURL"
: 
"
https://www.mendable.ai/
"

      }
    }
  ]
}

Scraping

Used to scrape a URL and get its content.

curl -X POST https://api.firecrawl.dev/v0/scrape \
    -H 
'
Content-Type: application/json
'
 \
    -H 
'
Authorization: Bearer YOUR_API_KEY
'
 \
    -d 
'
{

      "url": "https://mendable.ai"

    }
'

Response:

{
  
"success"
: 
true
,
  
"data"
: {
    
"content"
: 
"
Raw Content 
"
,
    
"markdown"
: 
"
# Markdown Content
"
,
    
"provider"
: 
"
web-scraper
"
,
    
"metadata"
: {
      
"title"
: 
"
Mendable | AI for CX and Sales
"
,
      
"description"
: 
"
AI for CX and Sales
"
,
      
"language"
: 
null
,
      
"sourceURL"
: 
"
https://www.mendable.ai/
"

    }
  }
}

Search (Beta)

Used to search the web, get the most relevant results, scrap each page and return the markdown.

curl -X POST https://api.firecrawl.dev/v0/search \
    -H 
'
Content-Type: application/json
'
 \
    -H 
'
Authorization: Bearer YOUR_API_KEY
'
 \
    -d 
'
{

      "query": "firecrawl",

      "pageOptions": {

        "fetchPageContent": true // false for a fast serp api

      }

    }
'
{
  
"success"
: 
true
,
  
"data"
: [
    {
      
"url"
: 
"
https://mendable.ai
"
,
      
"markdown"
: 
"
# Markdown Content
"
,
      
"provider"
: 
"
web-scraper
"
,
      
"metadata"
: {
        
"title"
: 
"
Mendable | AI for CX and Sales
"
,
        
"description"
: 
"
AI for CX and Sales
"
,
        
"language"
: 
null
,
        
"sourceURL"
: 
"
https://www.mendable.ai/
"

      }
    }
  ]
}

Intelligent Extraction (Beta)

Used to extract structured data from scraped pages.

curl -X POST https://api.firecrawl.dev/v0/scrape \
    -H 
'
Content-Type: application/json
'
 \
    -H 
'
Authorization: Bearer YOUR_API_KEY
'
 \
    -d 
'
{

      "url": "https://www.mendable.ai/",

      "extractorOptions": {

        "mode": "llm-extraction",

        "extractionPrompt": "Based on the information on the page, extract the information from the schema. ",

        "extractionSchema": {

          "type": "object",

          "properties": {

            "company_mission": {

                      "type": "string"

            },

            "supports_sso": {

                      "type": "boolean"

            },

            "is_open_source": {

                      "type": "boolean"

            },

            "is_in_yc": {

                      "type": "boolean"

            }

          },

          "required": [

            "company_mission",

            "supports_sso",

            "is_open_source",

            "is_in_yc"

          ]

        }

      }

    }
'
{
    
"success"
: 
true
,
    
"data"
: {
      
"content"
: 
"
Raw Content
"
,
      
"metadata"
: {
        
"title"
: 
"
Mendable
"
,
        
"description"
: 
"
Mendable allows you to easily build AI chat applications. Ingest, customize, then deploy with one line of code anywhere you want. Brought to you by SideGuide
"
,
        
"robots"
: 
"
follow, index
"
,
        
"ogTitle"
: 
"
Mendable
"
,
        
"ogDescription"
: 
"
Mendable allows you to easily build AI chat applications. Ingest, customize, then deploy with one line of code anywhere you want. Brought to you by SideGuide
"
,
        
"ogUrl"
: 
"
https://mendable.ai/
"
,
        
"ogImage"
: 
"
https://mendable.ai/mendable_new_og1.png
"
,
        
"ogLocaleAlternate"
: [],
        
"ogSiteName"
: 
"
Mendable
"
,
        
"sourceURL"
: 
"
https://mendable.ai/
"

      },
      
"llm_extraction"
: {
        
"company_mission"
: 
"
Train a secure AI on your technical resources that answers customer and employee questions so your team doesn't have to
"
,
        
"supports_sso"
: 
true
,
        
"is_open_source"
: 
false
,
        
"is_in_yc"
: 
true

      }
    }
}

Coming soon to the Langchain and LLama Index integrations.

Using Python SDK

Installing Python SDK

pip install firecrawl-py

Crawl a website

from
 firecrawl
 import
 FirecrawlApp


app
 =
 FirecrawlApp
(
api_key
=
"YOUR_API_KEY"
)

crawl_result
 =
 app
.
crawl_url
(
'mendable.ai'
, {
'crawlerOptions'
: {
'excludes'
: [
'blog/*'
]}})

# Get the markdown

for
 result
 in
 crawl_result
:
    
print
(
result
[
'markdown'
])

Scraping a URL

To scrape a single URL, use the scrape_url method. It takes the URL as a parameter and returns the scraped data as a dictionary.

url
 =
 'https://example.com'

scraped_data
 =
 app
.
scrape_url
(
url
)

Search for a query

Performs a web search, retrieve the top results, extract data from each page, and returns their markdown.

query
 =
 'What is Mendable?'

search_result
 =
 app
.
search
(
query
)

Contributing

We love contributions! Please read our contributing guide before submitting a pull request.

It is the sole responsibility of the end users to respect websites' policies when scraping, searching and crawling with Firecrawl. Users are advised to adhere to the applicable privacy policies and terms of use of the websites prior to initiating any scraping activities. By default, Firecrawl respects the directives specified in the websites' robots.txt files when crawling. By utilizing Firecrawl, you expressly agree to comply with these conditions.

- "漢字路" 한글한자자동변환 서비스는 교육부 고전문헌국역지원사업의 지원으로 구축되었습니다.
- "漢字路" 한글한자자동변환 서비스는 전통문화연구회 "울산대학교한국어처리연구실 옥철영(IT융합전공)교수팀"에서 개발한 한글한자자동변환기를 바탕하여 지속적으로 공동 연구 개발하고 있는 서비스입니다.
- 현재 고유명사(인명, 지명등)을 비롯한 여러 변환오류가 있으며 이를 해결하고자 많은 연구 개발을 진행하고자 하고 있습니다. 이를 인지하시고 다른 곳에서 인용시 한자 변환 결과를 한번 더 검토하시고 사용해 주시기 바랍니다.
- 변환오류 및 건의,문의사항은 juntong@juntong.or.kr로 메일로 보내주시면 감사하겠습니다. .
Copyright ⓒ 2020 By '전통문화연구회(傳統文化硏究會)' All Rights reserved.
 한국   대만   중국   일본