Skip to content

Node package for scraping Google Search Engine based on phantom and CasperJS

Notifications You must be signed in to change notification settings

majewan/node-google-scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

30 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Google Search Scraper

Google scraper is a simple tool to scrape google search engine writen for nodejs. Based on phantom and Casper. This work has been inspired by node-google-search-scraper package, and the usage is similar.

Features

  • Scrape Google results and can return rendered page for other usecase.
  • Handle old captchas :-(
  • Custom user agent and custom headers
  • Use JS for rendering Google
  • Promise or callback results

Usage

var GoogleSearchScraper = require('phantom-google-search-scraper');
GoogleSearchScraper.search({
  query : 'site:www.npmjs.com', // Query for google engine
  limit: 10, // Limit number of results
  keepPages: false, // Populate results.pages with rendered HTML content.
  solver: GoogleScraper.commandLineSolver, // Optional solver for resolving captcha (see commandLineSolver.js)
  userAgent: 'GoogleSearchScraper1.0',
  headers: { // Default http headers for PhantomJS
    'Accept-Language': 'ru-RU,en,*'
  },
  phantomOptions: [ // Command line options use for PhantomJS
    '--ignore-ssl-errors=yes'
  ]
})
.then(results => {
  console.log(results);
})
.catch(error => {
  console.error(error);
});

About

Node package for scraping Google Search Engine based on phantom and CasperJS

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published