guyueyingmu / avbook Star 9.3k Code Issues Pull requests AV ?影管理系?, avmoo , javbus , javlibrary 爬?,?上 AV 影片???,AV 磁力?接?据?,Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video Updated Mar 3, 2023 PHP
Tuhinshubhra / RED_HAWK Star 2.8k Code Issues Pull requests All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers wordpress crawler scanner http-header wordpress-scanner whois-lookup mx-lookup sql-scanner information-gathering geo-ip admin-scanner backups-finder sql-vulnerability-scannig subnet-lookup cms-detector cloudflare-detection subdomain-scanner reverse-ip-scan domain-authority-scanner page-authority-scanner Updated May 31, 2022 PHP
jae-jae / QueryList Star 2.6k Code Issues Pull requests ??? The progressive PHP crawler framework! ?雅的??式PHP采集?架。 crawler scraper spider querylist Updated Apr 30, 2024 PHP
spatie / crawler Sponsor Star 2.5k Code Issues Pull requests Discussions An easy to use, powerful crawler implemented in PHP. Can execute Javascript. php crawler guzzle concurrency Updated Mar 14, 2024 PHP
JayBizzle / Crawler-Detect Star 1.9k Code Issues Pull requests ?? CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent php crawler user-agent spider bots detect hacktoberfest Updated May 8, 2024 PHP
zorlan / skycaiji Star 1.9k Code Issues Pull requests ?天采集器是一款?源免?的爬?系?,?需点??????可采集?据,可?行在本地、??主机或云服?器中,?乎能采集所有?型的??,无??接各?CMS建站程序,免登????布?据,全自?无需人工干?!是??大?据采集?件中完全跨平台的云端爬?系? php crawler spider crawling webcrawler Updated May 15, 2024 PHP
diskoverdata / diskover-community Star 1.4k Code Issues Pull requests Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch python php metadata elasticsearch crawler storage filesystem duplicate-files disk-space file-indexing web-application file-manager filesystem-visualization duplicatefilefinder disk-usage disk-space-analyzer filesystem-analysis filesystem-indexer storage-analytics file-tagging Updated May 14, 2024 PHP
kiddyuchina / Beanbun Star 1.2k Code Issues Pull requests Beanbun 是用 PHP ??的多?程??爬??架,具有良好的?放性、高可?展性,基于 Workerman。 php crawler spider beanbun Updated Apr 10, 2023 PHP
duzun / hQuery.php Star 351 Code Issues Pull requests An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies. html fast php parser crawler scraper xml selectors html-parser xml-parser psr-4 css-selectors domcrawler psr-0 jquery-like jquery-selectors hquery broken-html invalid-html Updated Apr 20, 2024 PHP
yan68 / xcrawler Star 338 Code Issues Pull requests 快速、??且强大的PHP爬??架 php crawler scraper spider Updated Sep 12, 2018 PHP
crwlrsoft / crawler Star 300 Code Issues Pull requests Library for Rapid (Web) Crawler and Scraper Development php crawler scraper web-crawler scraping crawling web-scraper web-scraping scraping-websites web-crawling hacktoberfest Updated Mar 26, 2024 PHP
JayBizzle / Laravel-Crawler-Detect Star 299 Code Issues Pull requests A Laravel wrapper for CrawlerDetect - the web crawler detection library bot php crawler laravel spider detect Updated Jul 6, 2022 PHP
zrashwani / arachnid Star 254 Code Issues Pull requests Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites php crawler seo scraping Updated Sep 10, 2022 PHP
vezaynk / Sitemap-Generator-Crawler Star 236 Code Issues Pull requests PHP script to recursively crawl websites and generate a sitemap. Zero dependencies. php sitemap crawler xml seo sitemap-generator Updated Mar 2, 2023 PHP
spatie / robots-txt Sponsor Star 213 Code Issues Pull requests Determine if a page may be crawled from robots.txt, robots meta tags and robot headers php crawler robots-txt Updated May 14, 2024 PHP
webysther / packagist-mirror Sponsor Star 189 Code Issues Pull requests ???????? Create a mirror of packagist.org metadata for use locally with composer php crawler packagist composer mirror composer-packages packagist-mirror Updated Oct 8, 2022 PHP
subins2000 / search Sponsor Star 185 Code Issues Pull requests An Open Source Search Engine search php search-engine crawler Updated Feb 17, 2022 PHP
vormkracht10 / laravel-seo-scanner Sponsor Star 179 Code Issues Pull requests Scan your Laravel application routes for SEO improvements suggestions. crawler laravel scanner seo laravel-framework seotools seo-optimization seo-tools laravel-seo laravel-seo-scanner Updated May 2, 2024 PHP
hedii / php-crawler Star 134 Code Issues Pull requests A php crawler that finds emails on the internets php crawler vuejs laravel vue webscraper webcrawler webscraping php-crawler Updated May 20, 2021 PHP
Joel2B / XVideos-PornHub-RedTube-API Star 119 Code Issues Pull requests This script scrapes the HTML from different web pages to get the information from the video (XVideos, PornHub, RedTube) and you can use it in your own video player. php crawler video scrapper xvideos pornhub redtube pornhub-downloader pornhub-api scrapping-php pornhub-scraper Updated Aug 30, 2023 PHP