（更新）数据接口，小红书蒲公英，抖音巨量星图，快手磁力聚星，B站花火，腾讯广告互选，微博微任务，淘宝(带精确预售量、精确月销量)，拼多多，小红书，微信公众号，大众点评，快手，京东，饿了么，B站，知乎，微博，Bigo，TEMU，得物、贝壳，shopee，lazada, 百度指数，等数据接口；大模型训练预料

Use Github Action to automatically crawl bing daily wallpaper.

Crawler321: An intelligent and automatic crawler

Crawl a specific webpage, and returns all links to other pages contained in the page

Flexible Node.js AI-assisted crawler library

Playwright Spider Utils is a utility library for engineers using the Playwright framework to build web crawlers. This project provides common web scraping functions, simplifying the process of crawler development and enhancing productivity.

Ruby class to crawl a website using Nokogiri, MongoDB database, and MongoMapper ORM

Webcrawl is a Python web crawler that recursively follows links from a starting URL to extract and print unique HTTP links. Using 'requests and 'BeautifulSoup', it avoids revisits, handles errors, and supports configurable crawling depth. Ideal for gathering and analyzing web links.

一个极简单的爬取中国官方/主流媒体网站文章数据的应用，用户自定义检索关键词和爬取页数，在选定的目标网站模拟实时搜索进行爬取

浏览过的精彩逆向文章汇总，值得一看

Python GUI application for bulk image conversion. Converts images in a directory and its subdirectories to WebP (or PNG) format.

novel-plus 是一个多端（PC、WAP）阅读 、功能完善的小说 CMS 系统。包括小说推荐、小说检索、小说排行、小说阅读、小说书架、小说评论、小说爬虫、会员中心、作家专区、充值订阅、新闻发布等功能。

Crawl telegra.ph searching for nudes!

Scrapy dappy doo crawler for proxy sites

searching youtube comment by using Youtube API

The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

A Data Analysis and Recommendation system project on Netflix Movies and TV Series dataset with Python

Helps you to secure your whole site on the development time and admin pages from the Brute attack.

基于小红书 Web 端进行的请求封装。https://reajason.github.io/xhs/

Spider crawl entire net-works for fuicy files sitting on SMB shares. Search filenames or file content - regex supported!