Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inquiries on Experiment Reproduction #2

Closed
LeoYML opened this issue May 20, 2024 · 3 comments
Closed

Inquiries on Experiment Reproduction #2

LeoYML opened this issue May 20, 2024 · 3 comments

Comments

@LeoYML
Copy link

LeoYML commented May 20, 2024

This is an interesting and outstanding piece of work.
How can I reproduce the experiments detailed in the README?
Additionally, how does the performance compare when using GPT-4?

@holarissun
Copy link
Owner

holarissun commented May 24, 2024

Thanks for your interest in our work!

The experiments are done with GPT3.5 API --- combining different prompting prefixes or postfixes with the queries in different datasets (e.g., the GSM8K dataset).

There are some updates on "prompt optimization" after the very early stages when people tried to find the "magic words" as prompting strategies:

  1. According to a later paper by Google https://arxiv.org/abs/2309.03409
    Optimal prompts can be different for different types of LLMs.

    The take-away here is that the performance of the prompting strategy is LLM-dependent.

  2. This ICLR'24 paper https://arxiv.org/pdf/2309.06553 introduces a systematic way of discovering the optimal prompts for different queries.

    The take-away here is that the performance of the prompting strategy is query-dependent.

I hope this could help :)

Best,
Hao

@LeoYML
Copy link
Author

LeoYML commented May 24, 2024

Thank you very much for your quick and clear response.

@holarissun
Copy link
Owner

closing in active issue after 3 months :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants