i'm getting 54 sentences/s on inference with BERT on T4 GPU, is that good?

[renaud]

Question

I am getting around 54 sentences/s on inference for text classification.

What do you think? is that good? Does this compare with what you get?

Additional context

lang_model = "bert-base-cased"
do_lower_case = False
max_seq_len = 64
use_amp = None

• GCP
• 1 x T4 GPU
• n1-standard-2 (2 vCPUs, 7.5 GB memory)

[Timoeller]

Hey @renaud
we are currently doing a lot of inference benchmarking for Question Answering like described in deepset-ai/haystack/issues/39 where we also compare pytorch vs ONNX.

Concerning your throughput.
I think it is pretty slow - but I am not sure how a T4 GPU performs compared to the V100s we used.
One important paramter is the batch size. Did you test different batch size values?

Looking at Tanays post it takes 0.1621 seconds for a batch of size 64 to complete on a V100. That would make about 395 samples per sec. And this is done on QA, where the inference is much! more complex (lot of communication between GPU and CPU). Simple text classification should happen faster - my intuitive guess would be by a factor of 2-5 times...

Happy to interact here and make the textclassification inference faster together with you!

[Timoeller]

I did some speed benchmarking on text classification inference.
I used a V100 GPU and tested various batch size + max_seq_len values with inference on 1000 texts:

• seq len 256, (batch size, seconds)
  [(1, 15.71), (10, 5.47), (20, 5.31), (30, 5.31)]
• seq len 128, (batch size, seconds)
  [(1, 14.99), (10, 3.48), (20, 3.73), (30, 3.57)]

---

So deviding 1000/3.57 we get 280 samples/second for seq len=128 and batch size=30
I would suggest you should try increasing the batch size. A T4 will still be slower than a V100, but 54 samples/s is really low.
And I also realized that I might be wrong about textclassification inference being faster than QA inference - the numbers are comparable to a recent QA inference benchmark test.

[Timoeller]

Ok, why talking about intuition when one can also just check.

I tested QA inference vs Textclassification:

Text Classification on 1000 docs, max seq len 128
Batch size: 1, takes 14.947
Batch size: 3, takes 5.801
Batch size: 6, takes 3.904
Batch size: 10, takes 3.771
Batch size: 20, takes 3.758
Batch size: 30, takes 3.667

Question Answering on 1000 questions , max seq len 128 (doc + question just below 128 tokens)
Batch size: 1, takes 16.096
Batch size: 3, takes 6.172
Batch size: 6, takes 5.290
Batch size: 10, takes 4.951
Batch size: 20, takes 5.044
Batch size: 30, takes 4.930

So QA inference seems a bit slower than TextClassification inside FARM (0.4.4)

[Timoeller]

closing this now for inactivity, feel free to reopen