This repository contains code for the Elastic Open Web Crawler. Open Crawler enables users to easily ingest web content into Elasticsearch.

Open Crawler v0.1 is confirmed to be compatible with Elasticsearch v8.13.0 and above.

Indexing web content with the Open Crawler requires:

1. Running an instance of Elasticsearch (on-prem, cloud, or serverless)
2. Cloning of the Open Crawler repository (see Setup)
3. Configuring a crawler config file (see Configuring crawlers)
4. Using the CLI to begin a crawl job (see CLI commands)

Open Crawler runs crawl jobs on command based on config files in the config directory. Each URL endpoint found during the crawl will result in one document to be indexed into Elasticsearch.

Open Crawler performs crawl jobs in a multithreaded environment, where one thread will be used to visit one URL endpoint. The crawl results from these are added to a pool of results. These are indexed into Elasticsearch using the _bulk API once the pool reaches a configurable threshold.

A running instance of Elasticsearch is required to index documents into. If you don't have this set up yet, you can sign up for an Elastic Cloud free trial or check out the quickstart guide for Elasticsearch.

Open Crawler will attempt to use the _bulk API to index crawl results into Elasticsearch. To facilitate this connection, Open Crawler needs to have either an API key or a username/password configured to access the Elasticsearch instance. If using an API key, ensure that the API key has read and write permissions to access the index configured in output_index.

• Elasticsearch documentation for managing API keys for more details
• elasticsearch.yml.example file for all of the available Elasticsearch configurations for Crawler

POST /_security/api_key
{
  "name": "my-api-key",
  "role_descriptors": { 
    "my-crawler-role": {
      "cluster": ["all"],
      "indices": [
        {
          "names": ["my-crawler-index-name"],
          "privileges": ["all"]
        }
      ]
    }
  },
  "metadata": {
    "application": "my-crawler"
  }
}

Open Crawler has a Dockerfile that can be built and run locally.

1. Clone the repository: git clone https://github.com/elastic/crawler.git
2. Build the image docker build -t crawler-image .
3. Run the container docker run -i -d --name crawler crawler-image
   • -i allows the container to stay alive so CLI commands can be executed inside it
   • -d allows the container to run "detached" so you don't have to dedicate a terminal window to it
4. Confirm that CLI commands are working docker exec -it crawler bin/crawler version
   • Execute other CLI commands from outside of the container by prepending docker exec -it crawler <command>
5. Create a config file for your crawler. See Configuring crawlers for next steps. See Configuring crawlers for next steps.

See CONFIG.md for in-depth details on Open Crawler configuration files.

See DOCUMENT_SCHEMA.md for information regarding the Elasticsearch document schema and mappings.

Open Crawler does not have a graphical user interface. All interactions with Open Crawler take place through the CLI. When given a command, Open Crawler will run until the process is finished. OpenCrawler is not kept alive in any way between commands.

See CLI.md for a full list of CLI commands available for Crawler.