scrapy / scrapy Star 51k Code Issues Pull requests Scrapy, a fast high-level web crawling & scraping framework for Python. python crawler framework scraping crawling web-scraping hacktoberfest web-scraping-python Updated May 2, 2024 Python
iawia002 / lux Star 25.3k Code Issues Pull requests ?? Fast and simple video download library and CLI tool written in Go go golang crawler scraper downloader youtube video download tumblr bilibili qq youku iqiyi Updated May 3, 2024 Go
NaiboWang / EasySpider Sponsor Star 22.7k Code Issues Pull requests A visual no-code/code-free web crawler/spider易采集：一?可?化??器自?化??/?据采集/爬??件，可以无代??形化的??和?行爬?任?。?名：ServiceWrapper面向Web?用的智能化服?封?系?。 visualization html crawler scraper gui web spider frontend robotics parameters visual www data-collection batch-script batch-processing layman rpa visualprogramming code-free input-parameters Updated Apr 27, 2024 JavaScript
gocolly / colly Star 22.2k Code Issues Pull requests Elegant Scraper and Crawler Framework for Golang go golang crawler scraper framework spider scraping crawling Updated Apr 19, 2024 Go
jhao104 / proxy_pool Star 20.3k Code Issues Pull requests Python ProxyPool for web spider redis http crawler spider proxy Updated Feb 4, 2024 Python
binux / pyspider Star 16.3k Code Issues Pull requests A Powerful Spider(Web Crawler) System in Python. python crawler Updated Apr 30, 2024 Python
codelucas / newspaper Star 13.7k Code Issues Pull requests newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs: python crawler scraper news crawling news-aggregator Updated Apr 3, 2024 Python
shengqiangzhang / examples-of-web-crawlers Star 13.5k Code Issues Pull requests 一些非常有趣的python爬?例子,?新手比?友好,主要爬取淘?、天猫、微信、微信??、豆瓣、QQ等?站。(Some interesting examples of python crawlers that are friendly to beginners. ) python crawler spider example selenium multithreading stock wechat taobao pyquery tmall fund agent-pool wechat-report wereader Updated Dec 25, 2023 Python
apify / crawlee Star 12.2k Code Issues Pull requests Discussions Crawlee?A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation. nodejs javascript npm crawler scraper automation typescript web-crawler headless scraping crawling web-scraping web-crawling headless-chrome apify puppeteer playwright Updated May 3, 2024 TypeScript
code4craft / webmagic Star 11.2k Code Issues Pull requests A scalable web crawler framework for Java. java crawler framework scraping Updated Apr 23, 2024 Java
crawlab-team / crawlab Star 10.8k Code Issues Pull requests Discussions Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬?管理平台，支持任何?言和?架 go docker platform crawler spider web-crawler scrapy webcrawler scrapyd-ui webspider crawling-tasks crawlab spiders-management Updated Apr 19, 2024 Go
s0md3v / Photon Sponsor Star 10.5k Code Issues Pull requests Incredibly fast crawler designed for OSINT. python crawler osint spider information-gathering Updated Jan 4, 2024 Python
guyueyingmu / avbook Star 9.3k Code Issues Pull requests AV ?影管理系?， avmoo , javbus , javlibrary 爬?，?上 AV 影片???，AV 磁力?接?据?，Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video Updated Mar 3, 2023 PHP
injetlee / Python Star 9.3k Code Issues Pull requests Python脚本。模?登?知乎， 爬?，操作excel，微信公??，?程?机 python crawler excel wechat Updated Oct 10, 2023 Python
ssssssss-team / spider-flow Star 9.1k Code Issues Pull requests 新一代爬?平台，以?形化方式定?爬?流程，不?代??可完成爬?。 crawler spider web-crawler jsoup xpath webcrawler webspider web-spider spider-flow Updated Jun 14, 2023 Java
projectdiscovery / katana Star 8.7k Code Issues Pull requests Discussions A next-generation crawling and spidering framework. cli crawler headless web-spider spider-framework gocrawler Updated Apr 29, 2024 Go
Evil0ctal / Douyin_TikTok_Download_API Star 7k Code Issues Pull requests Discussions ??「Douyin_TikTok_Download_API」是一??箱?用的高性能?步?音、快手、TikTok、Bilibili?据爬取工具，支持API?用，在?批量解析及下?。 python api crawler scraper spider async aiohttp web-scraping asyncio asgi douyin tiktok fastapi tiktok-scraper pywebio no-watermark online-parsing douyin-tiktok-api douyin-tiktok-download douyin-scraper Updated May 2, 2024 Python
bda-research / node-crawler Star 6.6k Code Issues Pull requests Web Crawler/Spider for NodeJS + server-side jQuery ;-) nodejs javascript jquery crawler spider cheerio extract-data Updated Jan 4, 2024 JavaScript
lorien / awesome-web-scraping Star 6.3k Code Issues Pull requests List of libraries, tools and APIs for web scraping and data processing. crawler spider scraping crawling web-scraping captcha-recaptcha webscraping crawling-framework scraping-framework captcha-bypass scraping-tool crawling-tool scraping-python crawling-python Updated Apr 8, 2024 Makefile
BruceDone / awesome-crawler Star 6.1k Code Issues Pull requests A collection of awesome web crawler,spider in different languages crawler scraper awesome spider web-crawler web-scraper node-crawler Updated Apr 8, 2024
