Hi, @vblagoje thanks for the work! An issue recently opened by @danielbichuetti may also be related to this, which can be found here Issue #3289. It's possible that the multiprocessing in the FARMReader is preventing the multiprocessing in the Inferencer from working as expected. Edit: Sorry I misunderstood, the multiprocessing in the FARMReader and the Inferencer are the same.

Would it be possible to do the same CI performance check using the TransformerReader instead and see if this time difference is still observed? Edit: Actually, this isn't really relevant since the TransformerReader does not use the same Inferencer as the FARMReader so it will not have this multiprocessing performance problem.

Yeah totally-have a look at the notebook I made to measure FarmReader performance with/without multiprocessing. It should be relatively simple to create TransformerReader and inference it instead of FarmReader. I'll do it tomorrow. We should also do measurements on CPU as well for both versions.

@vblagoje Yeah sounds good! Also, I'm in the US right now hence why I'm still online so please feel free to respond tomorrow at a more reasonable time in Berlin!

@vblagoje See comment #3272 (comment) on the original issue opened by @Timoeller.

@vblagoje I think it is the correct move to remove multiprocessing from the inferencer since it seems to conflict with multiprocessing from the FastTokenizers in HF.

My only suggestion would be to go through infer.py and add deprecation warnings to all places where the variable multiprocessing_chunksize appears since it no longer used. For example in functions like inference_from_file, inference_from_objects, etc. in both the class Inferencer and the class QAInferencer.

@sjrl @masci let me know if we need to do anything else for this one