guyueyingmu / avbook Star 9.3k Code Issues Pull requests AV ?影管理系?, avmoo , javbus , javlibrary 爬?,?上 AV 影片???,AV 磁力?接?据?,Japanese Adult Video Library,Adult Video Magnet Links - Japanese Adult Video Database crawler scraper laravel database spider magnet-link guzzlehttp magnet adult javbus javlibrary avmoo adult-video Updated Jun 1, 2024 PHP
Tuhinshubhra / RED_HAWK Star 2.9k Code Issues Pull requests All in one tool for Information Gathering, Vulnerability Scanning and Crawling. A must have tool for all penetration testers wordpress crawler scanner http-header wordpress-scanner whois-lookup mx-lookup sql-scanner information-gathering geo-ip admin-scanner backups-finder sql-vulnerability-scannig subnet-lookup cms-detector cloudflare-detection subdomain-scanner reverse-ip-scan domain-authority-scanner page-authority-scanner Updated May 31, 2022 PHP
zorlan / skycaiji Star 1.9k Code Issues Pull requests ?天采集器是一款?源免?的爬?系?,?需点??????可采集?据,可?行在本地、??主机或云服?器中,?乎能采集所有?型的??,无??接各?CMS建站程序,免登????布?据,全自?无需人工干?!是??大?据采集?件中完全跨平台的云端爬?系? php crawler spider crawling webcrawler Updated May 15, 2024 PHP
jae-jae / QueryList Star 2.6k Code Issues Pull requests ??? The progressive PHP crawler framework! ?雅的??式PHP采集?架。 crawler scraper spider querylist Updated Apr 30, 2024 PHP
spatie / crawler Sponsor Star 2.5k Code Issues Pull requests Discussions An easy to use, powerful crawler implemented in PHP. Can execute Javascript. php crawler guzzle concurrency Updated Mar 14, 2024 PHP
JayBizzle / Crawler-Detect Star 1.9k Code Issues Pull requests ?? CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent php crawler user-agent spider bots detect hacktoberfest Updated Jun 7, 2024 PHP
kiddyuchina / Beanbun Star 1.2k Code Issues Pull requests Beanbun 是用 PHP ??的多?程??爬??架,具有良好的?放性、高可?展性,基于 Workerman。 php crawler spider beanbun Updated Apr 10, 2023 PHP
diskoverdata / diskover-community Star 1.4k Code Issues Pull requests Diskover Community Edition - Open source file indexer, file search engine and data management and analytics powered by Elasticsearch python php metadata elasticsearch crawler storage filesystem duplicate-files disk-space file-indexing web-application file-manager filesystem-visualization duplicatefilefinder disk-usage disk-space-analyzer filesystem-analysis filesystem-indexer storage-analytics file-tagging Updated Jun 7, 2024 PHP
subins2000 / search Sponsor Star 186 Code Issues Pull requests An Open Source Search Engine search php search-engine crawler Updated Feb 17, 2022 PHP
vezaynk / Sitemap-Generator-Crawler Star 237 Code Issues Pull requests PHP script to recursively crawl websites and generate a sitemap. Zero dependencies. php sitemap crawler xml seo sitemap-generator Updated Mar 2, 2023 PHP
tomasnorre / crawler Sponsor Star 55 Code Issues Pull requests Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc. php crawler extension typo3 indexer hacktoberfest cache-warmup Updated Jun 10, 2024 PHP
duzun / hQuery.php Star 352 Code Issues Pull requests An extremely fast web scraper that parses megabytes of invalid HTML in a blink of an eye. PHP5.3+, no dependencies. html fast php parser crawler scraper xml selectors html-parser xml-parser psr-4 css-selectors domcrawler psr-0 jquery-like jquery-selectors hquery broken-html invalid-html Updated Jun 12, 2024 PHP
webysther / packagist-mirror Sponsor Star 191 Code Issues Pull requests ???????? Create a mirror of packagist.org metadata for use locally with composer php crawler packagist composer mirror composer-packages packagist-mirror Updated Oct 8, 2022 PHP
hedii / php-crawler Star 133 Code Issues Pull requests A php crawler that finds emails on the internets php crawler vuejs laravel vue webscraper webcrawler webscraping php-crawler Updated May 20, 2021 PHP
zrashwani / arachnid Star 254 Code Issues Pull requests Crawl all unique internal links found on a given website, and extract SEO related information - supports javascript based sites php crawler seo scraping Updated Sep 10, 2022 PHP
yan68 / xcrawler Star 338 Code Issues Pull requests 快速、??且强大的PHP爬??架 php crawler scraper spider Updated Sep 12, 2018 PHP
lndj / Lcrawl Star 113 Code Issues Pull requests 一只?雅的正方??系?爬?。 php crawler exam grade zhengfang Updated Oct 14, 2019 PHP
Joel2B / XVideos-PornHub-RedTube-API Star 121 Code Issues Pull requests This script scrapes the HTML from different web pages to get the information from the video (XVideos, PornHub, RedTube) and you can use it in your own video player. php crawler video scrapper xvideos pornhub redtube pornhub-downloader pornhub-api scrapping-php pornhub-scraper Updated Jun 1, 2024 PHP
spatie / robots-txt Sponsor Star 213 Code Issues Pull requests Determine if a page may be crawled from robots.txt, robots meta tags and robot headers php crawler robots-txt Updated May 14, 2024 PHP
mmerian / phpcrawl Star 58 Code Issues Pull requests Copy of http://phpcrawl.cuab.de/ for using with composer php crawler composer phpcrawl Updated Jan 24, 2023 PHP