Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add alpca json inference #326

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

madhavajay
Copy link

The current inference is a little inflexible, I added something which lets you pass in json or a json file and either get json or a json file back out in the format of alpca style {"instruction":..."input":... "output"}.

I don't fully understand how the prompts are constructed so I think there are some changes there, but I have to test in the cloud due to no GPU on my Apple Silicon so it takes time to test.

For example:

$ accelerate launch scripts/finetune.py config-1b-qlora.yml --lora_model_dir="./qlora-output" --inference-json='{"input":"What is your name?"}' --inference-json-output=out.json
$ cat out.json
[{"input": "What is your name?", "output": "\"My Name Is... My name is Jessica, I am 32 years old.\""}]
Screenshot 2023-07-26 at 4 32 53 pm

@theobjectivedad
Copy link
Contributor

Good morning, just an inform - this is similar to a feature in #291

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants