OpenAI plugins connect ChatGPT to third-party applications. These plugins enable ChatGPT to interact with APIs defined by developers, enhancing ChatGPT's capabilities and allowing it to perform a wide range of actions (from openai documentation). A great example of such an implementation can be seen here.
For this project, we would like to take data captured from an Edge device (for example, audio, video, or sensor data captured at the Edge) and integrate it into an OpenAI plugin. This will allow developers to harness the power of ChatGPT in the cloud with custom data ingress at the Edge!
There is a wide range of potential use cases for this project, here are a few examples (generated with ChatGPT, of course).
Image Classification:
Plant and Crop Disease Detection: Edge devices with cameras can capture images of plants and crops, and the integrated LLM can identify diseases or pests present, along with suggested treatments.
Security and Surveillance: Edge devices can classify objects and activities in surveillance footage, generating real-time alerts or summaries of events for security personnel.
Health and Fitness: An LLM-equipped wearable device can classify exercise movements from video streams and provide real-time feedback on form and technique, acting as a virtual personal trainer.
Artwork Interpretation: An edge device equipped with an LLM can analyze images of artworks in a museum or gallery and provide natural language descriptions of the style, artist, historical context, and significance of the piece.
Audio Classification:
Environmental Sound Monitoring: Edge devices with microphones and LLMs can classify sounds in the environment, such as sirens, alarms, or machinery noises, and provide explanations or alerts based on the identified sound.
Language Learning: LLMs on language learning apps can classify pronunciation and speech patterns, providing learners with feedback on their accent, intonation, and fluency in real-time.
Emotion Analysis: An edge device equipped with a microphone can capture speech and use an LLM to classify the speaker's emotional state, enabling applications in mental health support and customer service.
Smart Appliances: Household appliances like refrigerators or ovens can use audio cues to trigger actions, such as classifying the sound of a door closing to determine when to start the cooling system or adjusting cooking times based on sizzling sounds.
Wildlife Monitoring: LLM-equipped devices can capture and classify sounds from the environment, helping researchers identify and track wildlife species based on their vocalizations.