Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Feature/huggingface pipeline streaming #331

Merged
merged 6 commits into from
Feb 15, 2024
Merged

Feature/huggingface pipeline streaming #331

Changes from 1 commit
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a61b28a
Added NeMo Guardrails streaming support for LLMs deployed using Huggi…
trebedea Feb 14, 2024
b50316e
GenerationChunk is needed for on_llm_new_token
trebedea Feb 14, 2024
65d714b
Added some additional documentation and example for streaming with Hu…
trebedea Feb 14, 2024
96a29d1
Merge branch 'develop' into feature/huggingface-pipeline-streaming
drazvan Feb 15, 2024
47e3f15
Disable dialog rails for HF dolly example.
drazvan Feb 15, 2024
634d15e
Tweak and fix the streaming for HuggingFace pipeline.
drazvan Feb 15, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
GenerationChunk is needed for on_llm_new_token
  • Loading branch information
@trebedea
trebedea committed Feb 14, 2024
commit b50316e40c3909853705c64304095a5111a2e098
4 changes: 3 additions & 1 deletion nemoguardrails/llm/providers/providers.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@
)
from langchain.llms.base import LLM
from langchain.llms.huggingface_pipeline import HuggingFacePipeline
from langchain.schema.output import GenerationChunk
from langchain_community import llms

from nemoguardrails.rails.llm.config import Model
Expand Down Expand Up @@ -116,8 +117,9 @@ async def _acall(
completion = ""
async for item in streamer:
completion += item
chunk = GenerationChunk(text=item)
if run_manager:
await run_manager.on_llm_new_token(item)
await run_manager.on_llm_new_token(item, chunk=chunk)
return completion

llm_result = await self._agenerate(
Expand Down