We read every piece of feedback, and take your input very seriously.
To see all available qualifiers, see our documentation.
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reproduce by
python -m sglang.launch_server --model-path databricks/dbrx-instruct --tp 8 --port 30000 --mem-frac 0.8 --enable-flashinfer
and
python3 bench_sglang.py --num-questions 10
The text was updated successfully, but these errors were encountered:
dbrx uses a gqa group size of 6, it should have been supported in flashinfer-ai/flashinfer#301 (and release v0.0.5)
Sorry, something went wrong.
Ying1123
No branches or pull requests
Reproduce by
and
The text was updated successfully, but these errors were encountered: