Skip to content

A list of common crawler user agents useful for retrieving metadata from links.

License

Notifications You must be signed in to change notification settings

plantaclaus/top-crawler-agents

Repository files navigation

top-crawler-agents

Last version NPM Status

A list of common crawler user agents useful for retrieving metadata from links
derivated from crawler-user-agents.

Some websites (such as Twitter) only return rich HTML metadata if you are identified as popular crawler (like Slack, or WhatsApp).

The list is derivated from crawler-user-agents after applying a script, so it's always up-to-date to latest changes.

Install

$ npm install top-crawler-agents --save

Usage

const uniqueRandomArray = require('unique-random-array')

const randomCrawlerAgent = uniqueRandomArray(require('top-crawler-agents'))

console.log(randomCrawlerAgent())
// => 'Slackbot-LinkExpanding (+https://api.slack.com/robots)'

Related

top-crawler-agents © Kiko Beats, released under the MIT License.
Authored and maintained by Kiko Beats with help from contributors.

kikobeats.com · GitHub Kiko Beats · Twitter @Kikobeats

About

A list of common crawler user agents useful for retrieving metadata from links.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • JavaScript 100.0%