Skip to content

Latest commit

 

History

History

web_scrapping_ai_agent

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
README.md
README.md
 
 
ai_scrapper.py
ai_scrapper.py
 
 
local_ai_scrapper.py
local_ai_scrapper.py
 
 
requirements.txt
requirements.txt
 
 

README.md

💻 Web Scrapping AI Agent

This Streamlit app allows you to scrape a website using OpenAI API and the scrapegraphai library. Simply provide your OpenAI API key, enter the URL of the website you want to scrape, and specify what you want the AI agent to extract from the website.

Features

  • Scrape any website by providing the URL
  • Utilize OpenAI's LLMs (GPT-3.5-turbo or GPT-4) for intelligent scraping
  • Customize the scraping task by specifying what you want the AI agent to extract

How to get Started?

  1. Clone the GitHub repository
git clone https://github.com/Shubhamsaboo/awesome-llm-apps.git
  1. Install the required dependencies:
pip install -r requirements.txt
  1. Get your OpenAI API Key
  • Sign up for an OpenAI account (or the LLM provider of your choice) and obtain your API key.
  1. Run the Streamlit App
streamlit run ai_scrapper.py

How it Works?

  • The app prompts you to enter your OpenAI API key, which is used to authenticate and access the OpenAI language models.
  • You can select the desired language model (GPT-3.5-turbo or GPT-4) for the scraping task.
  • Enter the URL of the website you want to scrape in the provided text input field.
  • Specify what you want the AI agent to extract from the website by entering a user prompt.
  • The app creates a SmartScraperGraph object using the provided URL, user prompt, and OpenAI configuration.
  • The SmartScraperGraph object scrapes the website and extracts the requested information using the specified language model.
  • The scraped results are displayed in the app for you to view