Skip to content
/ crawling-jobs Public template
forked from gal-dahan/crawling-jobs

Scans jobs in Israel By scraping public websites using a Node.js, every six hours.

Notifications You must be signed in to change notification settings

MatanP12/crawling-jobs

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

61 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Job Israel

Source code of Jobs-Israel

App Screenshot

Tech Stack

Client: React

Server: Node(v14.17.1), Express, Cheerio, Nodemailer

Database: MongoDB

Run Locally

Install project with npm.

Server:

    cd server
    npm install

Client:

    cd client 
    npm install

Start the server and client

npm Start

To set up your environment variables, create a new file named .env in the server folder of the project and copy the contents of .env.example into it. Then, replace the placeholders with your own values:

MONGO=mongodb+srv:https://username:[email protected]/your-db-name
[email protected]
MAIL_PASSWORD=your-email-password

Debugging

If you would like to change the timing of the CronJob for debugging purposes,or startup the project locally in first time to Start scanning and filling the database. you can modify the cronTime value in the crawling-jobs/server/server.js file. By default, the CronJob is set to run every 6 hours in production, but for debugging purposes, you may want to set it to run more frequently.

To change the timing of the CronJob, find the following code in the crawling-jobs/server/server.js:

const job = new CronJob({
  // 0 */6 * * * every 6 hours --- in production
  cronTime: "0 */6 * * *",
  onTick: function () {
    console.log("start crawling", new Date());
    startComeet();
    startWpComeet();
    startGreenHouse();
  },
  start: true,
  timeZone: 'Europe/Berlin'
});

To change the timing for debugging purposes, you can modify the cronTime value to run more frequently. For example, if you want the CronJob to run every minute, you can change the cronTime value to "* * * * *":

const job = new CronJob({
  // * * * * * every minute --- for Debugging
  cronTime: "* * * * *",
  onTick: function () {
    console.log("start crawling", new Date());
    startComeet();
    startWpComeet();
    startGreenHouse();
  },
  start: true,
  timeZone: 'Europe/Berlin'
});

About

Scans jobs in Israel By scraping public websites using a Node.js, every six hours.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • JavaScript 100.0%