Skip to content

ray-project/llmperf-leaderboard

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

5 Commits
 
 
 
 
 
 
 
 

Repository files navigation

LLMPerf Leaderboard 🏆

Utilizing the LLMPerf, we have benchmarked a selection of LLM inference providers. Our analysis focuses on evaluating their performance, reliability, and efficiency under the following key metrics:

  • Output tokens throughput, which represents the average number of output tokens returned per second. This metric is important for applications that require high throughput, such as summarization and translation, and easy to compare across different models and providers.
  • Time to first token (TTFT), which represents the duration of time that LLM returns the first token. TTFT is especially important for streaming applications, such as chatbots.

The LLMPerf Leaderboard displays results in a clear, transparent manner. Our aim is to provide users and developers with vital insights into the capabilities and limitations of each provider, informing decisions for future integrations and deployments. In line with our commitment to transparency and utility, we also provide reproducible steps in Run Configurations as shown below:

Run Configurations

For each of the benchmark run, it is performed with the below command template from the LLMPerf repository

   python token_benchmark_ray.py \
    --model <MODEL_NAME> \
    --mean-input-tokens 550 \
    --stddev-input-tokens 0 \
    --mean-output-tokens 150 \
    --stddev-output-tokens 0 \
    --max-num-completed-requests 150 \
    --num-concurrent-requests 5 \
    --llm-api <litellm/openai> 

For each provider, we perform:

  • Total number of requests: 150
  • Concurrency: 5 (5 concurrent requests to the provider)
  • Prompt's token length: 550
  • Expected output length: 150
  • Tested models: 7B, 13B, and 70B of LLama-2 chat models

We ran the LLMPerf clients on an AWS EC2 (Instance type: i4i.large) from us-west-2 (Oregon) region. The results were up-to-date of December 19, 2023, 3am PST. You could find the detailed results in the raw_data folder.

Caveats and Disclaimers

Note that there may be some possible source of biases or discrepancies from your perceived behavior:

  • The endpoints provider backend might vary widely, so this is not a reflection on how the software runs on a particular hardware.
  • The results may vary with time of day.
  • The results (e.g. measurement of TTFT) depend on client location, and can also be biased by some providers lagging on the first token in order to increase ITL. Our current measurement location is from us-west (Oregon).
  • The results is only a proxy of the system capabilities and is also impacted by the existing system load and provider traffic.
  • The results may not correlate with users’ workloads.

Output Tokens Throughput (tokens/s)

The output tokens throughput is measured as the average number of output tokens returned per second. We collect results by sending 150 requests to each LLM inference provider, and calculate the mean output tokens throughput based on 150 requests. A higher output tokens throughput indicates a higher throughput of the LLM inference provider.

70B Models

Framework Model Median Mean Min Max P25 P75 P95 P99
anyscale meta-llama/Llama-2-70b-chat-hf 66 63 22 86 56 72 77 82
bedrock meta.llama2-70b-chat-v1 21 21 13 22 20 22 22 22
fireworks accounts/fireworks/models/llama-v2-70b-chat 40 40 33 46 38 42 45 46
groq llama2-70b-4096 185 184 148 208 174 195 207 208
lepton llama2-70b 33 33 31 39 32 34 34 38
perplexity llama-2-70b-chat 30 30 8 44 29 31 36 44
replicate meta/llama-2-70b-chat 10 9 2 11 10 10 11 11
together together_ai/togethercomputer/llama-2-70b-chat 65 64 25 79 61 68 74 76

13B Models

Framework Model Median Mean Min Max P25 P75 P95 P99
anyscale meta-llama/Llama-2-13b-chat-hf 120 120 81 156 110 128 141 148
bedrock meta.llama2-13b-chat-v1 36 35 19 39 33 38 38 39
fireworks accounts/fireworks/models/llama-v2-13b-chat 42 42 39 45 41 43 44 44
lepton llama2-13b 43 43 37 48 42 44 46 48
replicate meta/llama-2-13b-chat 16 18 6 35 12 20 35 35
together together_ai/togethercomputer/llama-2-13b-chat 102 101 1 123 98 108 119 122

7B Models

Framework Model Median Mean Min Max P25 P75 P95 P99
anyscale meta-llama/Llama-2-7b-chat-hf 51 51 45 57 49 54 56 57
fireworks accounts/fireworks/models/llama-v2-7b-chat 76 76 53 82 75 78 79 82
lepton llama2-7b 36 36 33 40 35 38 40 40
replicate meta/llama-2-7b-chat 26 32 2 78 20 35 73 77
together together_ai/togethercomputer/llama-2-7b-chat 75 75 50 95 70 81 87 90

Time to First Token (seconds)

For streaming applications, the TTFT is how long before the LLM returns the first token.

70B Models

Framework Model Median Mean Min Max P25 P75 P95 P99
anyscale meta-llama/Llama-2-70b-chat-hf 0.21 0.25 0.18 0.59 0.19 0.30 0.36 0.47
bedrock meta.llama2-70b-chat-v1 0.39 0.41 0.29 0.72 0.37 0.41 0.54 0.69
fireworks accounts/fireworks/models/llama-v2-70b-chat 0.51 0.51 0.32 0.96 0.39 0.56 0.79 0.95
groq llama2-70b-4096 0.22 0.23 0.17 0.36 0.19 0.24 0.3 0.35
lepton llama2-70b 0.93 0.9 0.72 1.12 0.82 0.96 1.01 1.1
perplexity llama-2-70b-chat 0.37 0.42 0.29 0.70 0.34 0.52 0.63 0.66
replicate meta/llama-2-70b-chat 1.19 5.08 0.97 71.57 1.03 1.7 24.23 63.63
together together_ai/togethercomputer/llama-2-70b-chat 0.63 0.62 0.46 0.89 0.55 0.67 0.77 0.87

13B Models

Framework Model Median Mean Min Max P25 P75 P95 P99
anyscale meta-llama/Llama-2-13b-chat-hf 0.20 0.22 0.18 0.56 0.19 0.22 0.34 0.50
bedrock meta.llama2-13b-chat-v1 0.27 0.33 0.16 0.77 0.25 0.3 0.74 0.76
fireworks accounts/fireworks/models/llama-v2-13b-chat 0.49 0.47 0.28 0.66 0.39 0.54 0.59 0.65
lepton llama2-13b 1.08 1.07 0.82 1.4 0.95 1.15 1.24 1.37
replicate meta/llama-2-13b-chat 5.65 6.27 0.98 17.01 3.62 8.31 14.76 16.71
together together_ai/togethercomputer/llama-2-13b-chat 0.54 0.89 0.39 0.91 0.46 0.60 0.70 0.81

* Perplexity doesn't offer 13B models when the data was gathered. More details for models offered could be found here.

7B Models

Framework Model Median Mean Min Max P25 P75 P95 P99
anyscale meta-llama/Llama-2-7b-chat-hf 0.20 0.23 0.18 0.50 0.19 0.23 0.34 0.46
fireworks accounts/fireworks/models/llama-v2-7b-chat 0.33 0.33 0.21 1.09 0.32 0.34 0.37 0.88
lepton llama2-7b 1.13 1.11 0.88 1.33 1.04 1.18 1.29 1.32
replicate meta/llama-2-7b-chat 3.68 3.61 0.99 7.2 2.31 5.01 6.37 6.99
together together_ai/togethercomputer/llama-2-7b-chat 0.52 0.58 0.42 0.95 0.46 0.71 0.84 0.94

* Perplexity doesn't offer Llama-2-7B models when the data was gathered. More details for models offered could be found here.

* Bedrock doesn't offer Llama-2-7B models when the data was gathered. More details for models offered could be found here.

Feedback

  • Kindly provide your feedback using the link. We would love to hear from you.
  • For LLM inference service providers interested in having their API featured on this dashboard, please submit an issue or reach out to us via email for further communication (e.g. setting up of test accounts and etc).

About

No description, website, or topics provided.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published