Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Semantic Search for Detections #11899

Merged
merged 16 commits into from
Jun 21, 2024
Merged

Semantic Search for Detections #11899

merged 16 commits into from
Jun 21, 2024

Conversation

hunterjm
Copy link
Contributor

@hunterjm hunterjm commented Jun 12, 2024 •
edited
Loading

Now that 0.14 beta is out, I've re-implemented #8998 using the new ZeroMQ brokers. This PR is strictly for the backend to enable this feature. A separate PR will be opened for the frontend implementation.

While some things below have changed, a general description of the implementation plan can be found in #8980.

This Backend PR

  • Add a "description" to the events table data field
  • Create a new config entry for semantic_search and genai
    • Able to configure whether or not it is enabled, default disabled
    • Able to configure and enable multiple different GenAI providers for descriptions
      • Google Gemini
      • OpenAI
      • Ollama
  • Create a new app service for embeddings
    • Instantiate a ChromaDB collection using CLIP
    • On startup, load existing ended event thumbnails into Chroma if they do not yet exist
      • Implemented via .reindex file in the /config directory. It takes a while depending on the number of events in the DB... this is on a 12 core CPU: Embedded 10987 thumbnails and 851 descriptions in 730.0870163440704 seconds
      • Recommend just letting build up index naturally as new events come in and old ones are removed.
    • Implement a maintainer for embeddings to add ended detection images and descriptions to ChromaDB

Follow-up Frontend PR

  • Create an HTTP API for POST /api/events/<id>/description allowing external systems creating descriptions
  • Add ChromaDB Logs to the frontend viewer
  • Implement "Search" page in Frontend
    • Search box for text similarity
    • Additional filters on metadata (date range, camera, labels, etc)
    • Implement HTTP API on backend modifying /events endpoint to allow for similarity search as well
    • Ability to edit descriptions in the frontend
      • Maybe via an info icon that pulls up a right bar like the logs page?
    • Thumbnail similarity, pulling up detections with very close thumbnails
      • Could also live in right bar

@netlify Netlify
Copy link

netlify bot commented Jun 12, 2024 •
edited
Loading

✅ Deploy Preview for frigate-docs ready!

Name Link
🔨 Latest commit 89937e2
🔍 Latest deploy log https://app.netlify.com/sites/frigate-docs/deploys/666e6e247d19ce000894d430
😎 Deploy Preview https://deploy-preview-11899--frigate-docs.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

hunterjm
hunterjm commented Jun 12, 2024
View reviewed changes
Copy link
Contributor Author

@hunterjm hunterjm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments for additional context around some decisions.

frigate/__main__.py Show resolved Hide resolved
frigate/comms/zmq_proxy.py Show resolved Hide resolved
frigate/events/cleanup.py Outdated Show resolved Hide resolved
blakeblackshear
blakeblackshear reviewed Jun 14, 2024
View reviewed changes
Copy link
Owner

@blakeblackshear blakeblackshear left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this looks great as usual with your contributions.

frigate/embeddings/embeddings.py Outdated Show resolved Hide resolved
frigate/comms/events_updater.py Outdated Show resolved Hide resolved
frigate/embeddings/maintainer.py Outdated Show resolved Hide resolved
frigate/embeddings/maintainer.py Show resolved Hide resolved
frigate/embeddings/maintainer.py Outdated Show resolved Hide resolved
frigate/events/cleanup.py Outdated Show resolved Hide resolved
@hunterjm
Copy link
Contributor Author

All of the initial review comments should now be resolved.

@NickM-27 NickM-27 changed the base branch from dev to 0.14.1 June 21, 2024 20:40
@hunterjm hunterjm mentioned this pull request Jun 21, 2024
Merged
@NickM-27 NickM-27 merged commit e99f879 into blakeblackshear:0.14.1 Jun 21, 2024
6 checks passed
@hunterjm hunterjm deleted the semantic_search_backend branch June 21, 2024 21:34
hunterjm added a commit to hunterjm/frigate that referenced this pull request Jun 24, 2024
@hunterjm
Semantic Search for Detections (blakeblackshear#11899)
4255a65 
* Initial re-implementation of semantic search

* put docker-compose back and make reindex match docs

* remove debug code and fix import

* fix docs

* manually build pysqlite3 as binaries are only available for x86-64

* update comment in build_pysqlite3.sh

* only embed objects

* better error handling when genai fails

* ask ollama to pull requested model at startup

* update ollama docs

* address some PR review comments

* fix lint

* use IPC to write description, update docs for reindex

* remove gemini-pro-vision from docs as it will be unavailable soon

* fix OpenAI doc available models

* fix api error in gemini and metadata for embeddings
NickM-27 pushed a commit that referenced this pull request Jul 2, 2024
@hunterjm @NickM-27
Semantic Search for Detections (#11899)
a007873 
* Initial re-implementation of semantic search

* put docker-compose back and make reindex match docs

* remove debug code and fix import

* fix docs

* manually build pysqlite3 as binaries are only available for x86-64

* update comment in build_pysqlite3.sh

* only embed objects

* better error handling when genai fails

* ask ollama to pull requested model at startup

* update ollama docs

* address some PR review comments

* fix lint

* use IPC to write description, update docs for reindex

* remove gemini-pro-vision from docs as it will be unavailable soon

* fix OpenAI doc available models

* fix api error in gemini and metadata for embeddings
NickM-27 pushed a commit that referenced this pull request Jul 9, 2024
@hunterjm @NickM-27
Semantic Search for Detections (#11899)
7d499b5 
* Initial re-implementation of semantic search

* put docker-compose back and make reindex match docs

* remove debug code and fix import

* fix docs

* manually build pysqlite3 as binaries are only available for x86-64

* update comment in build_pysqlite3.sh

* only embed objects

* better error handling when genai fails

* ask ollama to pull requested model at startup

* update ollama docs

* address some PR review comments

* fix lint

* use IPC to write description, update docs for reindex

* remove gemini-pro-vision from docs as it will be unavailable soon

* fix OpenAI doc available models

* fix api error in gemini and metadata for embeddings
NickM-27 pushed a commit that referenced this pull request Jul 9, 2024
@hunterjm @NickM-27
Semantic Search for Detections (#11899)
176a9e5 
* Initial re-implementation of semantic search

* put docker-compose back and make reindex match docs

* remove debug code and fix import

* fix docs

* manually build pysqlite3 as binaries are only available for x86-64

* update comment in build_pysqlite3.sh

* only embed objects

* better error handling when genai fails

* ask ollama to pull requested model at startup

* update ollama docs

* address some PR review comments

* fix lint

* use IPC to write description, update docs for reindex

* remove gemini-pro-vision from docs as it will be unavailable soon

* fix OpenAI doc available models

* fix api error in gemini and metadata for embeddings
NickM-27 pushed a commit that referenced this pull request Jul 13, 2024
@hunterjm @NickM-27
Semantic Search for Detections (#11899)
52d9386 
* Initial re-implementation of semantic search

* put docker-compose back and make reindex match docs

* remove debug code and fix import

* fix docs

* manually build pysqlite3 as binaries are only available for x86-64

* update comment in build_pysqlite3.sh

* only embed objects

* better error handling when genai fails

* ask ollama to pull requested model at startup

* update ollama docs

* address some PR review comments

* fix lint

* use IPC to write description, update docs for reindex

* remove gemini-pro-vision from docs as it will be unavailable soon

* fix OpenAI doc available models

* fix api error in gemini and metadata for embeddings
spanner3003 pushed a commit to spanner3003/frigate that referenced this pull request Jul 13, 2024
@hunterjm @spanner3003
Semantic Search for Detections (blakeblackshear#11899)
f74653a 
* Initial re-implementation of semantic search

* put docker-compose back and make reindex match docs

* remove debug code and fix import

* fix docs

* manually build pysqlite3 as binaries are only available for x86-64

* update comment in build_pysqlite3.sh

* only embed objects

* better error handling when genai fails

* ask ollama to pull requested model at startup

* update ollama docs

* address some PR review comments

* fix lint

* use IPC to write description, update docs for reindex

* remove gemini-pro-vision from docs as it will be unavailable soon

* fix OpenAI doc available models

* fix api error in gemini and metadata for embeddings
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@blakeblackshear blakeblackshear blakeblackshear approved these changes

@NickM-27 NickM-27 NickM-27 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@hunterjm @blakeblackshear @NickM-27