Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/colang nb changes #394

Merged
merged 20 commits into from
Mar 13, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
20 commits
Select commit Hold shift + click to select a range
e5a21c8
Add missing core library flow variables
schuellc-nvidia Feb 29, 2024
d19c216
Fix standard library issue in state tracking flows
schuellc-nvidia Mar 4, 2024
ab64dce
Replace some log statements in standard library with print statement
schuellc-nvidia Mar 4, 2024
07518a6
Add import_paths to ignore list for config merging
schuellc-nvidia Mar 4, 2024
75784a8
Expose all event arguments as attributes of events
schuellc-nvidia Mar 4, 2024
d23a784
Fix some more flow names in standard library
schuellc-nvidia Mar 4, 2024
d56622c
Add file name to parsing error
schuellc-nvidia Mar 6, 2024
be85f1c
Change some log to print statement for better visibility
schuellc-nvidia Mar 6, 2024
59df028
Fix action lifetime when shared in concurrent flows
schuellc-nvidia Mar 7, 2024
c402508
Expose action context variables directly as action attributes
schuellc-nvidia Mar 8, 2024
b29676f
Add new example about how to call a custom llm prompt using a python …
schuellc-nvidia Mar 8, 2024
ad3915c
Remove deprecated test
schuellc-nvidia Mar 11, 2024
2a728e7
Update core flow library
schuellc-nvidia Mar 11, 2024
ee632ea
Update Colang core library
schuellc-nvidia Mar 11, 2024
b024ec8
Some simple renaming
schuellc-nvidia Mar 12, 2024
338ef65
Generate proper action events in chat CLI
schuellc-nvidia Mar 12, 2024
9822569
Make colang core library accessible for other colang scripts
schuellc-nvidia Mar 12, 2024
74d5783
Update Colang 2.0 VSCode code highlighting extension
schuellc-nvidia Mar 13, 2024
8942372
Formatting fix.
drazvan Mar 13, 2024
7c01f77
Merge branch 'develop' into feature/colang-nb-changes
drazvan Mar 13, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update Colang core library
  • Loading branch information
schuellc-nvidia committed Mar 12, 2024
commit ee632ea143cf18cd6a936cc07348661300c76803
50 changes: 25 additions & 25 deletions tests/test_configs/multi_modal_demo_v2_x/core_flow_library.co
Original file line number Diff line number Diff line change
Expand Up @@ -293,7 +293,7 @@ flow bot suggest $text
# DON'T CHANGE! Currently, hard-wired with LLM prompt generation
# -----------------------------------

flow tracking of bot talking state
flow tracking bot talking state
# meta: loop_id=state_tracking
global $bot_talking_state
if $user_talking_state == None
Expand All @@ -303,7 +303,7 @@ flow tracking of bot talking state
await bot said something
$bot_talking_state = False

flow tracking of user talking state
flow tracking user talking state
# meta: loop_id=state_tracking
global $user_talking_state
if $user_talking_state == None
Expand All @@ -313,15 +313,15 @@ flow tracking of user talking state
await user said something
$user_talking_state = False

flow tracking of unhandled user intent state
flow tracking unhandled user intent state
# meta: loop_id=state_tracking
global $user_intent_state
when unhandled user intent as $flow
$unhandled_user_intent_state = $flow.context.flow_name
orwhen FlowFinished(flow_id=r"^user ") as $event
$unhandled_user_intent_state = None

flow tracking of visual choice selection state
flow tracking visual choice selection state
# meta: loop_id=state_tracking
global $choice_selection_state
when VisualChoiceSceneAction.Started()
Expand All @@ -334,7 +334,7 @@ flow tracking of visual choice selection state
orwhen VisualChoiceSceneAction.Finished()
$choice_selection_state = None

flow tracking of user utterance state
flow tracking user utterance state
# meta: loop_id=state_tracking
global $last_user_transcript
global $last_user_message
Expand Down Expand Up @@ -370,14 +370,14 @@ flow wait $time_s $timer_id="wait_timer_{{uid()}}"
# meta: loop_id=NEW
await TimerBotAction(timer_name=$timer_id, duration=$time_s)

flow catching of colang errors
flow catching colang error
"""A flow to catching of any runtime Colang errors"""
# meta: loop_id=catch_colang_errors
match ColangError() as $event
print "Colang error: {{$event.error_type}} - {{escape($event.error)}}"
bot say "Excuse me, what did you say?"

flow catching of undefined flows
flow catching start of undefined flows
"""A flow to catching of the start of undefined flows."""
priority 0.1
match UnhandledEvent(event="StartFlow") as $event
Expand All @@ -386,7 +386,7 @@ flow catching of undefined flows
# We need to abort the flow that sent the FlowStart event since it might be waiting for it
send StopFlow(flow_instance_uid=$event.source_flow_instance_uid)

flow catching of unexpected user utterance
flow catching unexpected user utterance
"""A flow to catching of unhandled user utterances."""
priority 0.1
match UnhandledEvent(event="UtteranceUserActionFinished") as $event
Expand All @@ -404,7 +404,7 @@ flow await_flow_by_name $flow_name
match FlowStarted(flow_id=$flow_name, flow_start_uid=$new_flow_start_uid) as $event_ref
match $event_ref.flow.Finished()

flow interruption handling bot talking $mode="inform"
flow handling bot talking interruption $mode="inform"
"""Handling the bot talking interruption reaction."""
# meta: loop_id=bot_interruption
user interrupted bot talking
Expand All @@ -420,7 +420,7 @@ flow interruption handling bot talking $mode="inform"
# LLM mechanics
# ----------------------------------

flow poll llm request response $interval
flow polling llm request response $interval
# meta: loop_id=llm_response_polling
match StartGenerateUserIntentAction() as $event_ref
or StartGenerateFlowContinuationAction() as $event_ref
Expand All @@ -433,13 +433,13 @@ flow poll llm request response $interval
send $polling_timer.Stop()
send $posture.Stop()

flow trigger user intent for unhandled user utterance
flow generating user intent for unhandled user utterance
"""This is the fallback flow that takes care of unhandled user utterances and will generate a user intent."""
global $bot_talking_state
match UnhandledEvent(event="UtteranceUserActionFinished", loop_ids={$loop_id}) as $event
if $bot_talking_state == False
$transcript = $event.final_transcript
log "trigger user intent for unhandled user utterance: {{$transcript}}"
log "generating user intent for unhandled user utterance: {{$transcript}}"
$action = 'user said "{{$transcript}}"'
$intent = await derive user intent from user action $action 20

Expand Down Expand Up @@ -480,14 +480,14 @@ flow generate then continue interaction
# Experimental flows
# ----------------------------------

flow respond to unhandled user intent
flow responding to unhandled user intent
"""This is the fallback flow that takes care of unhandled user utterances and will generate a user intent."""
await unhandled user intent
$flow_name = await generate interaction continuation
await await_flow_by_name $flow_name
await RemoveFlowsAction(flow_ids=[$flow_name])

flow handle undefined flow starts
flow handling start of undefined flow
"""We want to start an undefined flow."""

match UnhandledEvent(event="StartFlow") as $event
Expand Down Expand Up @@ -541,20 +541,20 @@ flow custom instructions
# --------------------------------------
# Posture Management Flows
# --------------------------------------
flow manage listening posture
# meta: loop_id=manage listening posture
flow managing listening posture
# meta: loop_id=managing listening posture
user started saying something
start bot posture "listening"
match UtteranceUserAction.Finished()

flow manage talking posture
# meta: loop_id=manage talking posture
flow managing talking posture
# meta: loop_id=managing talking posture
bot started saying something
start bot posture "talking"
bot said something

flow manage thinking posture
# meta: loop_id=manage thinking posture
flow managing thinking posture
# meta: loop_id=managing thinking posture
global $bot_talking_state

match UtteranceUserAction.Finished()
Expand All @@ -564,11 +564,11 @@ flow manage thinking posture
start bot posture "thinking"
bot started saying something

flow manage bot postures
# meta: loop_id=manage bot postures
flow managing bot postures
# meta: loop_id=managing bot postures
activate track bot talking state
activate manage listening posture
activate manage thinking posture
activate manage talking posture
activate managing listening posture
activate managing thinking posture
activate managing talking posture
start bot posture "attentive"
wait indefinitely
36 changes: 18 additions & 18 deletions tests/test_configs/multi_modal_demo_v2_x/demo.co
Original file line number Diff line number Diff line change
Expand Up @@ -197,16 +197,16 @@ flow check limitations with user


flow showcase selector
activate tracking of visual choice selection state
activate tracking visual choice selection state

$showcase_selection_ui = None
global $bot_talking_state

while True
log "while loop restarted"
if $showcase_selection_ui == None
activate trigger user intent for unhandled user utterance
activate interruption handling bot talking "inform"
activate generating user intent for unhandled user utterance
activate handling bot talking interruption "inform"
start VisualChoiceSceneAction(prompt= "Pick a showcase", support_prompts=["You can click on any option below","Or just say the 'Option C'","Or you can say 'I want to play the guessing game'"],choice_type="selection", allow_multiple_choices=False, options= [{"id": "game", "text": "A: Guessing Game", "image":"number 2 horizontal orientation"}, {"id": "multimodality", "text": "B: Multimodality", "image":"newton s cradle horizontal orientation"}, {"id": "backchanneling", "text": "C: Backchanneling", "image":"channel mixer horizontal orientation"}, {"id": "posture", "text": "D: Posture modulation", "image":"gauge horizontal orientation"}, {"id": "proactive", "text": "E: Proactive turn-taking", "image":"speech bubble horizontal orientation"}]) as $showcase_selection_ui
start bot inform "Please pick one of the showcases."

Expand All @@ -228,40 +228,40 @@ flow showcase selector

orwhen user picked number guessing game showcase
send $showcase_selection_ui.Stop()
send FinishFlow(flow_id="trigger user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="interruption handling bot talking", deactivate=True)
send FinishFlow(flow_id="generating user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="handling bot talking interruption", deactivate=True)
$showcase_selection_ui = None
bot inform "Great! You picked the number guessing game!"
showcase number guessing game

orwhen user picked multimodality showcase
send $showcase_selection_ui.Stop()
send FinishFlow(flow_id="trigger user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="interruption handling bot talking", deactivate=True)
send FinishFlow(flow_id="generating user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="handling bot talking interruption", deactivate=True)
$showcase_selection_ui = None
bot inform "Great! You picked the multimodality showcase!"
showcase multimodality

orwhen user picked backchanneling showcase
send $showcase_selection_ui.Stop()
send FinishFlow(flow_id="trigger user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="interruption handling bot talking", deactivate=True)
send FinishFlow(flow_id="generating user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="handling bot talking interruption", deactivate=True)
$showcase_selection_ui = None
bot inform "Great! You picked the backchanneling example!"
showcase backchannelling interaction

orwhen user picked posture showcase
send $showcase_selection_ui.Stop()
send FinishFlow(flow_id="trigger user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="interruption handling bot talking", deactivate=True)
send FinishFlow(flow_id="generating user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="handling bot talking interruption", deactivate=True)
$showcase_selection_ui = None
bot inform "Great! You picked the posture showcase!"
showcase posture capabilities

orwhen user picked proactive showcase
send $showcase_selection_ui.Stop()
send FinishFlow(flow_id="trigger user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="interruption handling bot talking", deactivate=True)
send FinishFlow(flow_id="generating user intent for unhandled user utterance", deactivate=True)
send FinishFlow(flow_id="handling bot talking interruption", deactivate=True)
$showcase_selection_ui = None
bot inform "Great! You picked the proactive turn-taking example!"
showcase proactive turn taking
Expand All @@ -278,12 +278,12 @@ flow showcase selector

flow main
# meta: exclude from llm
#activate catching of unexpected user utterance
#activate catching unexpected user utterance
activate ignored_utterance_action_bugfix
activate catching of undefined flows
activate catching of colang errors
activate poll llm request response 1.0
activate tracking of bot talking state
activate catching start of undefined flows
activate catching colang error
activate polling llm request response 1.0
activate tracking bot talking state
#activate faq

start scene show textual information $title="Welcome to the Tech Demo of Colang 2.0" $text="" $header_image="https://blogs.nvidia.com/wp-content/uploads/2023/04/NeMo-Guardrails-KV-x1280.jpg" as $welcome_ui
Expand Down
6 changes: 3 additions & 3 deletions tests/test_configs/mvp_v2_x_d/config.co
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,7 @@ flow fallback
match FlowStarted(flow_id=$flow_info['name']) as $event_ref
match $event_ref.flow.Finished()

flow handle undefined flow starts
flow handling start of undefined flow
"""If we reach a point where we need to start a flow that does not exist, we generate it."""
# meta: exclude from llm

Expand All @@ -149,7 +149,7 @@ flow handle undefined flow starts

if not $exists
$flow_id = $event.flow_id
log "'handle undefined flow starts' -> generate new flow '{{$event.flow_id}}'"
log "'handling start of undefined flow' -> generate new flow '{{$event.flow_id}}'"
$flow_source = await GenerateFlowFromNameAction(name=$flow_id)
await AddFlowsAction(config=$flow_source)
send StartFlow(flow_id=$flow_id)
Expand All @@ -175,7 +175,7 @@ flow main
activate custom instructions
activate unknown intent
activate fallback
activate handle undefined flow starts
activate handling start of undefined flow
# activate question
# activate tell the user the weather

Expand Down