stream_select warning FD_SETSIZE

[PDFCoder]

We run into a problem on our live server (debian, PHP 7.1) with phpmailer 6.0.6 – everything has been working fine there with version 5.2.x.

If we try to send an email (can even be a simple text, or the smtp_check of the sample files!) we get a php warning and the server is running into a timeout:

PHP Warning: stream_select(): You MUST recompile PHP with a larger value of FD_SETSIZE.
It is set to 1024, but you have descriptors numbered at least as high as 1314.
--enable-fd-setsize=2048 is recommended, but you may want to set it
to equal the maximum number of open files supported by your system,
in order to avoid seeing this error again at a later date. in /home/www/xxx/phpmailer/src/SMTP.php on line 1125

Our hoster won't recompile PHP, so the suggested solution won't work.

We made some futher tests and found out, that the stream_select if condition in line 1125 didn't exist in phpmailer 5.2.x. If we delete this condition, everythings seems to work fine again (as far as we could test it now; with bigger mails, attachments).

if (!stream_select($selR, $selW, $selW, $this->Timelimit)) { …

If we set the timelimit of the function to a small value (like 3 seconds) the server loops a warning to the error log until the script is terminated:
PHP Warning: stream_select(): No stream arrays were passed in /home/www/xxx/phpmailer/src/SMTP.php

So what is the idea behind the stream_select condition? Is there a fix to this problem?

On other servers/accounts (same setup) the problem does not appear …!? Any idea?

[PDFCoder]

OK, as I see now, this might be a duplicate to #1618.

But do you see any disadvantages in deleting the stream_select if condition? As I understand, it should prevent a timeout under certain circumstances?

[Synchro]

This doesn't require recompiling PHP, but it does require being able to alter runtime file descriptor limits, usually via sysctl and ulimit, which you may also not have access to, but your ISP may be able to alter that. The origin of this use of stream select is here. You can see that it replaced a simpler timeout mechanism that failed in other ways. You could create a child class and override that usage (e.g. by reverting to the old implementation) as a workaround, but be aware that doing that may create other problems elsewhere.

[dpeca]

Very interesting thing is that:

• When PHP is running on Apache (mod_php) - bug occur
• When PHP is running on PHP-FPM - there is no bug
• When PHP is running from SSH console - there is no bug

Tested on the same machine.
Debian9, php packages from sury.org

[dpeca]

And, btw, in sysctl.conf we have
fs.file-max = 200000
And in limits.conf we have

* soft     nofile         200000
* hard     nofile         200000

So file descriptors ar not limited at all...

[dpeca]

Btw, solution that @PDFCoder found (to comment if (!stream_select) solved a problem.

[Synchro]

I'd recommend double-checking what the apache user's file limits actually are, for example run sudo -u www-data bash and then ulimit -n. IME it's common to find large global settings, but then to find that individual processes/users have the default 1024 handle limit.

[chriscroome]

I've come across this issue and it doesn't appear to depend on the user that is running Apache and their file limits, it also doesn't appear to depend on PHP, Apache and OpenSSL being recompiled to support more than 1024 file descriptors, I have tried to address this by re-building the Debian packages and on a server running these rebuilt packages the test script posted to this issue can be run and will open more that 1024 sockets if edited to do so (it won't open more than 1024 sockets with the versions of Apache and PHP Debian provides).

This Plesk guide suggests that libc-client2002edebian also needs rebuilding, this is now the libc-client2007e package and it provides the uw-imap library, I couldn't get this to recompile, could this be where the problem lies?

[dpeca]

@Synchro

root@server:~# sudo -u www-data bash
www-data@server:/root$ ulimit -n
200000

[Synchro]

OK, so it's not that. Can you see if your PHP is built with its own FD limit, as @PDFCoder mentioned?

[dpeca]

I'm pretty sure we would have much larger issues if ulimit is only 1024...

@Synchro
How I can check it?

[chriscroome]

This is how I checked the enable-fd-setsize variable on Debian Stretch:

apt install -y php-dev
php-config --configure-options | sed 's/\s\+/\n/g' | grep enable-fd-setsize

But having it set to a value higher than 1024 doesn't actually appear to help…

[dpeca]

I got just empty output with it, @chriscroome

[dpeca]

# php-config7.0 --configure-options
--includedir=/usr/include --mandir=/usr/share/man --infodir=/usr/share/info --disable-silent-rules --libdir=/usr/lib/x86_64-linux-gnu --libexecdir=/usr/lib/x86_64-linux-gnu --disable-maintainer-mode --disable-dependency-tracking --prefix=/usr --enable-cli --disable-cgi --disable-phpdbg --with-config-file-path=/etc/php/7.0/cli --with-config-file-scan-dir=/etc/php/7.0/cli/conf.d --build=x86_64-linux-gnu --host=x86_64-linux-gnu --config-cache --cache-file=/build/php7.0-7.0.33/config.cache --libdir=${prefix}/lib/php --libexecdir=${prefix}/lib/php --datadir=${prefix}/share/php/7.0 --program-suffix=7.0 --sysconfdir=/etc --localstatedir=/var --mandir=/usr/share/man --disable-all --disable-debug --disable-rpath --disable-static --with-pic --with-layout=GNU --without-pear --enable-filter --with-openssl=yes --with-pcre-regex=/usr --enable-hash --with-mhash=/usr --enable-libxml --enable-session --with-system-tzdata --with-zlib=/usr --with-zlib-dir=/usr --enable-dtrace --enable-pcntl --with-libedit=shared,/usr build_alias=x86_64-linux-gnu host_alias=x86_64-linux-gnu CFLAGS=-g -O2 -fdebug-prefix-map=/build/php7.0-7.0.33=. -fstack-protector-strong -Wformat -Werror=format-security -O2 -Wall -pedantic -fsigned-char -fno-strict-aliasing -g

There is no enable-fd-setsize

[chriscroome]

Empty output means it has the default of 1024 as far as I could work out.

[chriscroome]

I you have a dev / test Stretch server feel free to try the debs I built which are available here https://deb.webarch.net/ they have --enable-fd-setsize 2048 but servers running them still hit the limit, it appears to make no difference…

[Synchro]

I'm not sure at what point the limit is applied - for example if it's per-process and you're running under a non-threaded SAPI like mod_php, you're less likely to run into it than if you're using a threaded single process like PHP-FPM.

I'd also expect that if you're running into a limit like this, it's likely to occur in at least slightly random random places, rather than consistently the same place. It would be useful to get someone with PHP internals knowledge on this.

[dpeca]

Both Apache (with mod_php) and PHP-FPM are configured to switch process UID and GID to user - so we do not run scripts as www-data, for sure.
For Apache we are using mod_ruid2 to switch UID and GID to script owner user...
I'm testing PHPMailer as user that only runs phpmailer test script.

And as I previously said, there is no issues if script is run through PHP-FPM.
Hard to believe that mod_ruid2 is not doing proper UID and GID switching... we are running 50 servers with mod_ruid2, everything works fine with switching process to UID and GID of script owner...

Again, I expect pretty much issues if scripts are reaching ulimits... but we don't have any issues, even we hosts 200 sites on that server.
This is the first time to see error about FD_SETSIZE...

[chriscroome]

I'm using Apache 2 ITK MPM, it is in Debian, I haven't come across mod_ruid2 before, I see that this is also in Debian, I guess they do more-or-less that same thing…

[eldy]

Same trouble here on ubuntu 18.04, apache2 and thousands of virtual hosts using their own UID and GID with mpm_itk.
The limit is reached only when using phpmailer. Other applications and other features are working correctly.
Note that the trouble occurs when doing an action as simple as sending 1 email and only 1 email from 1 instance. It is strange that the library needs to open so many files, just to send 1 email. The default limit of 1024 was never a trouble in several years of activity.
Nevermind, I confirm that solution that @PDFCoder found (to comment if (!stream_select) solved the problem on our side too.

[Synchro]

It's not the library opening "so many" files. It's the library opening 1 file, and PHP not having enough file descriptors available to let it happen.

[mj-mehdizadeh]

this was helped me
https://bugs.php.net/bug.php?id=37025

[benjamin-lieser]

The problem is caused because of a stream_select() call in PHPMailer. The select syscall is discouraged because of the limitations: https://man7.org/linux/man-pages/man2/select.2.html . As long as php uses the select syscall to implement the stream_select() function this will be a problem.
In PHPMailer we do not need the full functionality because it's just waiting on one socket.
Is there a reason why against a simple blocking socket with socket_set_block and socket_read?
Right now this makes version 6 unusable in some environments. And also makes it slower in case the FD_SETSIZE is high enough.

[Synchro]

I don't know the internals implications of the stream_select, but I do recall that it was introduced in PHPMailer to avoid a different bug in #604.

If you have a better and more reliable way to implement this, PRs are welcome.

[kylex]

We're also running up against this issue, and would love to see a solution that doesn't involve having to recompile PHP.

[Synchro]

As mentioned earlier in this thread, you can revert to the old behaviour (complete with its accompanying, but different bugs) by providing your own child class that overrides just the get_lines method, and how you do that is shown in this doc.

[benjamin-lieser]

As mentioned earlier in this thread, you can revert to the old behaviour (complete with its accompanying, but different bugs) by providing your own child class that overrides just the get_lines method, and how you do that is shown in this doc.

Are we sure the old version is still buggy, because it was correct php, but there were problems in the php implementation, which might have been fixed.
It's hard to produce something which works in all cases, because we have to build on a really subpar php implementation around the io systemcalls

[Synchro]

Could be, but I don't know enough about PHP internals to judge. It could do with someone who actually knows rather than guessing. Also remember we still maintain compatibility back to PHP 5.5.