Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add Tests for extend generation logic #1184

Open
wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Add Tests for extend generation logic #1184

Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
6cc6a37
feat(dspy): in extend_generation, compare key values to None, not to
Jun 19, 2024
2b55ca3
feat(dspy): add tests for extend generation
Jun 19, 2024
d12e0f2
feat(dspy): revert changes to extend generation logic
Jun 19, 2024
ea29442
feat(dspy): rm redundant comments
Jun 19, 2024
15aef81
feat(dspy): revert changes to test file writing
Jun 19, 2024
4771535
feat(dspy): changed assertions so tests pass and added conversation logs
Jun 20, 2024
bcab220
feat(dsp): clarify what is generated in each generation
Jun 20, 2024
20fc91b
Merge branch 'stanfordnlp:main' into mipro-v2-extend-generation
mikeedjones Jun 21, 2024
7ec499b
Merge branch 'main' into mipro-v2-extend-generation
mikeedjones Jul 9, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
feat(dsp): clarify what is generated in each generation
  • Loading branch information
JONEMI19 committed Jun 20, 2024
commit bcab220171d5f02fbc2833bce649f296519cf3bb
129 changes: 15 additions & 114 deletions tests/predict/test_predict.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -254,77 +254,44 @@ def test_extend_generation(SandwichIdea):
dspy.settings.configure(lm=lm)

prediction = Predict(SandwichIdea)(meal="lunch", dietary_requiements="N/A")

# The logged conversation:
# The logged conversation (additional newlines removed, [..] indicates the generation):
# === DummyLM ===
# Based on the meal and dietary requirements, suggest a sandwich idea.

# ---

# Follow the following format.

# Meal: ${meal}

# Dietary Requiements: ${dietary_requiements}

# Bread: ${bread}

# Protein: ${protein}

# Fat: ${fat}

# Garnish: ${garnish}

# Sauce: ${sauce}

# ---

# Meal: lunch

# Dietary Requiements: N/A

# Bread: whole wheat

# Bread: [whole wheat
# Protein: turkey

# Fat: avocado
# Fat: avocado]
# ===
# === DummyLM ===
# Based on the meal and dietary requirements, suggest a sandwich idea.

# ---

# Follow the following format.

# Meal: ${meal}

# Dietary Requiements: ${dietary_requiements}

# Bread: ${bread}

# Protein: ${protein}

# Fat: ${fat}

# Garnish: ${garnish}

# Sauce: ${sauce}

# ---

# Meal: lunch

# Dietary Requiements: N/A

# Bread: whole wheat

# Protein: turkey

# Fat: avocado

# Garnish: tomato

# Sauce: mustard
# Garnish: [tomato
# Sauce: mustard]
# ===

assert prediction.bread == "whole wheat"
Expand All @@ -342,79 +309,45 @@ def test_extend_generation_rolled_back_when_field_is_skipped(SandwichIdea):
]
)
dspy.settings.configure(lm=lm)

# The logged conversation:
# The logged conversation (additional newlines removed, [..] indicates the generation):
# === DummyLM ===
# Based on the meal and dietary requirements, suggest a sandwich idea.

# ---

# Follow the following format.

# Meal: ${meal}

# Dietary Requiements: ${dietary_requiements}

# Bread: ${bread}

# Protein: ${protein}

# Fat: ${fat}

# Garnish: ${garnish}

# Sauce: ${sauce}

# ---

# Meal: lunch

# Dietary Requiements: N/A

# Bread: white

# Bread:[ white
# Fat: butter

# Garnish: lettuce

# Sauce: mayo
# Sauce: mayo]
# ===
# === DummyLM ===
# Based on the meal and dietary requirements, suggest a sandwich idea.

# ---

# Follow the following format.

# Meal: ${meal}

# Dietary Requiements: ${dietary_requiements}

# Bread: ${bread}

# Protein: ${protein}

# Fat: ${fat}

# Garnish: ${garnish}

# Sauce: ${sauce}

# ---

# Meal: lunch

# Dietary Requiements: N/A

# Bread: white Fat: butter Garnish: lettuce Sauce: mayo

# Protein: ham

# Protein:[ ham
# Fat: butter

# Garnish: lettuce

# Sauce: mayo
# Sauce: mayo]
# ===

predictor = Predict(SandwichIdea)(meal="lunch", dietary_requiements="N/A")
Expand All @@ -433,76 +366,44 @@ def test_extend_generation_with_empty_field(SandwichIdea):
]
)
dspy.settings.configure(lm=lm)
# The logged conversation:
# The logged conversation (additional newlines removed, [..] indicates the generation):
# === DummyLM ===
# Based on the meal and dietary requirements, suggest a sandwich idea.

# ---

# Follow the following format.

# Meal: ${meal}

# Dietary Requiements: ${dietary_requiements}

# Bread: ${bread}

# Protein: ${protein}

# Fat: ${fat}

# Garnish: ${garnish}

# Sauce: ${sauce}

# ---

# Meal: lunch

# Dietary Requiements: N/A

# Bread: white

# Bread:[ white
# Protein:

# Fat: butter

# Garnish: lettuce
# Garnish: lettuce]
# ===
# === DummyLM ===
# Based on the meal and dietary requirements, suggest a sandwich idea.

# ---

# Follow the following format.

# Meal: ${meal}

# Dietary Requiements: ${dietary_requiements}

# Bread: ${bread}

# Protein: ${protein}

# Fat: ${fat}

# Garnish: ${garnish}

# Sauce: ${sauce}

# ---

# Meal: lunch

# Dietary Requiements: N/A

# Bread: white

# Protein: Fat: butter Garnish: lettuce

# Fat: lettuce

# Sauce: mayo
# Fat:[ lettuce
# Sauce: mayo]
# ===

predictor = Predict(SandwichIdea)(meal="lunch", dietary_requiements="N/A")
Expand Down
Loading