Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
-
Updated
Jul 6, 2024 - Go
Distributed web crawler admin platform for spiders management regardless of languages and frameworks. 分布式爬虫管理平台,支持任何语言和框架
A Unix-style personal search engine and web crawler for your digital footprint.
This program provides efficient web scraping services for Tor and non-Tor sites. The program has both a CLI and REST API.
Golang Webcrawler for Elasticsearch
🍛 Curry é um WebCrawler escrito em Golang com finalidade de verificar o valor do câmbio de Dólar para Real (USDxBRL) em algumas lojas no Paraguay.
webpage crawler and mini search engine
Sherlock Gopher | Sherlock Gopher is a distributed System which will crawl over a given website and analyse the given HTML.
Golang implementation of the disk repository with update management (DRUM) framework as presented by Hsin-Tsang Lee, Derek Leonard, Xiaoming Wang, and Dmitri Loguinov in the paper "IRLbot: Scaling to 6 Billion Pages and Beyond"
🖨 Scrapes GCODE commands and documentation from numerous 3d printer firmware websites.
Implementing my own web crawler in golang
Crawl web pages, convert asset URL's into raw source code, and embed the source code back into the DOM.
Complete project to Crawl and Scrape SEO data from any sitemap.xml and save data into CSV and JSON file.
Add a description, image, and links to the webcrawler topic page so that developers can more easily learn about it.
To associate your repository with the webcrawler topic, visit your repo's landing page and select "manage topics."