Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bump file pool size #14966

Merged
merged 1 commit into from May 14, 2021
Merged

Bump file pool size #14966

merged 1 commit into from May 14, 2021

Conversation

ghost
Copy link

@ghost ghost commented May 13, 2021

Reason no 1: Performance:
https://www.libtorrent.org/tuning.html#file-pool

libtorrent keeps an LRU file cache. Each file that is opened, is stuck in the cache. The main purpose of this is because of anti-virus software that hooks on file-open and file close to scan the file. Anti-virus software that does that will significantly increase the cost of opening and closing files. However, for a high performance seed, the file open/close might be so frequent that it becomes a significant cost. It might therefore be a good idea to allow a large file descriptor cache. Adjust this though settings_pack::file_pool_size

Reason no 2: Potential I/O errors:
The error:
#14962 (comment)

Most likely cause:
#14962 (comment)

So basically when the file was closed, some other application(most likely antivirus) opened it for exclusive access, thus resulting in I/O errors when qBt tried to re-open it. If the file pool was large enough there'd be no need to close and reopen files so frequently as to cause collition with other apps.

P.S - I'm not sure what might be a good number for this settings. But I've seen many people use this number(5000) when they tune libtorrent. If keeping too many file descriptors open doesn't incur a much higher cost in comparison to gained performance then it might be worth it to bump this.

@glassez glassez added the Core label May 13, 2021
@glassez
Copy link
Member

glassez commented May 13, 2021

Hope it doesn't exceed any system limits...

@ghost
Copy link
Author

ghost commented May 13, 2021

Some systems may have lower file descriptor limits. But in windows I haven’t heard of any limitations. If this exceeds system limits then qBt would be limited to that number instead by the kernal without any undesired effects.

@ghost
Copy link
Author

ghost commented May 13, 2021

We can make this change limited to Windows only if we’re concerned about hitting file descriptor limits and failing to keep files/connections open.

@xavier2k6
Copy link
Member

xavier2k6 commented May 13, 2021

We can make this change limited to Windows only if we’re concerned about hitting file descriptor limits and failing to keep files/connections open.

Just to chime in here - my file_pool_size is set at 8,000 & has been for a very long time (windows 10)

Have seen issues before where just raising file_pool_size to say 50 helped (I think it really depends on a users hardware/config)

Example

file_pool_size = 4,000
#12914 (comment)

I believe it will also help when closing qbittorrent - the process seems to exit/disappear quicker in task manager

The accessing/updating of the resume files for users who say don't have anti-virus may be have been impacted by this default setting all along............just a thought!

EDIT:
ref #12825 (comment) onwards

@arvidn
Copy link
Contributor

arvidn commented May 13, 2021

I believe windows has a much higher default limit for number of file handles, than Linux and MacOS. It might make sense to have different defaults for different systems.

Opening files with exclusive access isn't really a problem on non-windows either.

@xavier2k6
Copy link
Member

I believe windows has a much higher default limit for number of file handles, than Linux and MacOS. It might make sense to have different defaults for different systems.

Opening files with exclusive access isn't really a problem on non-windows either.

for reference:

#12825 (comment)

@FranciscoPombal
Copy link
Member

@glassez

Hope it doesn't exceed any system limits...

At least on Linux, qBittorrent has been setting this to "unlimited" for a while now anyway: #10853.

Copy link
Member

@FranciscoPombal FranciscoPombal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As others have mentioned, I too have seen instances where bumping this definitely helped. Combined with the fact that on Windows there is practically no limit, and the fact that other systems don't really have issues due to opening with exclusive access, I think this is a welcome change.

@glassez glassez requested a review from Chocobo1 May 13, 2021 10:32
@glassez glassez added this to the 4.4.0 milestone May 13, 2021
@Chocobo1 Chocobo1 merged commit ea3b897 into qbittorrent:master May 14, 2021
@Chocobo1
Copy link
Member

@An0n666
Thank you!

@ghost ghost deleted the filepoolbump branch May 14, 2021 05:05
IceCodeNew pushed a commit to IceCodeNew/qBittorrent-Enhanced-Edition that referenced this pull request May 30, 2021
glassez pushed a commit to glassez/qBittorrent that referenced this pull request Jul 3, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants