guyueyingmu / avbook Star 9.3k Code Issues Pull requests AV ?影管理系?, avmoo , javbus , javlibrary 爬?,?上 AV 影片???,AV 磁力?接?据?,Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video Updated Jun 1, 2024 PHP
Tuhinshubhra / RED_HAWK Star 2.9k Code Issues Pull requests All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers wordpress crawler scanner http-header wordpress-scanner whois-lookup mx-lookup sql-scanner information-gathering geo-ip admin-scanner backups-finder sql-vulnerability-scannig subnet-lookup cms-detector cloudflare-detection subdomain-scanner reverse-ip-scan domain-authority-scanner page-authority-scanner Updated May 31, 2022 PHP
jae-jae / QueryList Star 2.6k Code Issues Pull requests ??? The progressive PHP crawler framework! ?雅的??式PHP采集?架。 crawler scraper spider querylist Updated Apr 30, 2024 PHP
spatie / crawler Sponsor Star 2.5k Code Issues Pull requests Discussions An easy to use, powerful crawler implemented in PHP. Can execute Javascript. php crawler guzzle concurrency Updated Mar 14, 2024 PHP
JayBizzle / Crawler-Detect Star 1.9k Code Issues Pull requests ?? CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent php crawler user-agent spider bots detect hacktoberfest Updated Jun 7, 2024 PHP
zorlan / skycaiji Star 1.9k Code Issues Pull requests ?天采集器是一款?源免?的爬?系?,?需点??????可采集?据,可?行在本地、??主机或云服?器中,?乎能采集所有?型的??,无??接各?CMS建站程序,免登????布?据,全自?无需人工干?!是??大?据采集?件中完全跨平台的云端爬?系? php crawler spider crawling webcrawler Updated May 15, 2024 PHP
diskoverdata / diskover-community Star 1.4k Code Issues Pull requests Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch python php metadata elasticsearch crawler storage filesystem duplicate-files disk-space file-indexing web-application file-manager filesystem-visualization duplicatefilefinder disk-usage disk-space-analyzer filesystem-analysis filesystem-indexer storage-analytics file-tagging Updated Jun 7, 2024 PHP
kiddyuchina / Beanbun Star 1.2k Code Issues Pull requests Beanbun 是用 PHP ??的多?程??爬??架,具有良好的?放性、高可?展性,基于 Workerman。 php crawler spider beanbun Updated Apr 10, 2023 PHP
duzun / hQuery.php Star 352 Code Issues Pull requests An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies. html fast php parser crawler scraper xml selectors html-parser xml-parser psr-4 css-selectors domcrawler psr-0 jquery-like jquery-selectors hquery broken-html invalid-html Updated Jun 12, 2024 PHP
yan68 / xcrawler Star 338 Code Issues Pull requests 快速、??且强大的PHP爬??架 php crawler scraper spider Updated Sep 12, 2018 PHP
crwlrsoft / crawler Star 304 Code Issues Pull requests Library for Rapid (Web) Crawler and Scraper Development php crawler scraper web-crawler scraping crawling web-scraper web-scraping scraping-websites web-crawling hacktoberfest Updated Jun 5, 2024 PHP
JayBizzle / Laravel-Crawler-Detect Star 301 Code Issues Pull requests A Laravel wrapper for CrawlerDetect - the web crawler detection library bot php crawler laravel spider detect Updated Jul 6, 2022 PHP
zrashwani / arachnid Star 254 Code Issues Pull requests Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites php crawler seo scraping Updated Sep 10, 2022 PHP
vezaynk / Sitemap-Generator-Crawler Star 237 Code Issues Pull requests PHP script to recursively crawl websites and generate a sitemap. Zero dependencies. php sitemap crawler xml seo sitemap-generator Updated Mar 2, 2023 PHP
spatie / robots-txt Sponsor Star 213 Code Issues Pull requests Determine if a page may be crawled from robots.txt, robots meta tags and robot headers php crawler robots-txt Updated May 14, 2024 PHP
webysther / packagist-mirror Sponsor Star 191 Code Issues Pull requests ???????? Create a mirror of packagist.org metadata for use locally with composer php crawler packagist composer mirror composer-packages packagist-mirror Updated Oct 8, 2022 PHP
subins2000 / search Sponsor Star 187 Code Issues Pull requests An Open Source Search Engine search php search-engine crawler Updated Feb 17, 2022 PHP
vormkracht10 / laravel-seo-scanner Sponsor Star 180 Code Issues Pull requests Scan your Laravel application routes for SEO improvements suggestions. crawler laravel scanner seo laravel-framework seotools seo-optimization seo-tools laravel-seo laravel-seo-scanner Updated May 2, 2024 PHP
hedii / php-crawler Star 133 Code Issues Pull requests A php crawler that finds emails on the internets php crawler vuejs laravel vue webscraper webcrawler webscraping php-crawler Updated May 20, 2021 PHP
blogdaren / PHPCreeper Star 121 Code Issues Pull requests A new generation of multi-process async event-driven spider engine based on workerman. Support headless browser. http://www.phpcreeper.com socket crawler spider asynchronous high-performance multi-process event-driven workerman proxy-pool Updated Jun 11, 2024 PHP