Skip to content

Ekultek/elastic-custom-scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

46 Commits
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

3K Toolkit

Tailored Data Collection for Eurasia

gcloud components install docker-credential-gcr cloud-build-local beta alpha kubectl

Folder Structure

├── .gitignore
├── README.md
├── pyvenv.cfg
├── requirements.txt
├── results
│   ├── json
│   │   └── example.json
│   └── text
│       └── example.txt
└── src
    ├── dorks
    │   ├── BINARYEDGE.md
    │   └── SHODAN.md
    ├── elastic
    │   ├── 00_MAINSCRAPER.py
    │   ├── README.md
    │   └── keywords.txt
    ├── elasticdump
    │   ├── README.md
    │   ├── download_indices.sh
    │   └── index_names.txt
    ├── kibana
    │   ├── DataScraperKibana.py
    │   ├── DataSearchKibana.py
    │   └── README.md
    ├── miscprojects
    │   ├── crawler.py
    │   ├── requester.py
    │   └── shodan-utils.py
    └── scrapy
        ├── scraper
        │   ├── __init__.py
        │   ├── items.py
        │   ├── middlewares.py
        │   ├── pipelines.py
        │   ├── settings.py
        │   └── spiders
        │       ├── __init__.py
        │       └── bgpview.py
        └── scrapy.cfg
  1. Useful software, divided into different programs based on utility, is available under the src file.

  2. git pull should take care of any updates most of the time. Run git stash if having any issues. Be aware local changes will be overwritten.

  3. Remember to run pip3 install -r requirements.txt. If feeling special also spin up a virtualenv.

  4. README.md can be found in every software folder for each utility.

  5. Explanation of Utilities:

    • Dorks: useful queries for Shodan and BinaryEdge
    • Elastic: scraper for all elastic databases based on keywords
    • Elasticdump: download from a cluster selected indices as JSON
    • Kibana: [draft] kibana scraper
    • Miscprojects: [draft]
    • Scrapy: [draft] advanced web/spider crawler
    • Social: [draft] Instagram and Twitter scraping / analysis tools

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published