Mail() function not working correctly in PHP 8.x

[hansherlighed]

Description

The following code:

<?php
mail("[email protected]", "This is the subject", "This is the body");
?>

Resulted in this output of the DATA part of the communication from tcpdump on the receiving SMTP server when building container "FROM php:8.0.14-fpm-alpine" or "FROM php:fpm-alpine":

Received: by f3de6efeb3a0 (sSMTP sendmail emulation); Sat, 12 Feb 2022 06:28:10 +0100\r\n
From: "Linux User" <www-data@f3de6efeb3a0>\r\n
Date: Sat, 12 Feb 2022 06:28:10 +0100\r\n
To: [email protected]\r
\r\n
Subject: This is the subject\r
\r\n
\r
\r\n
This is the Body\r
\r\n
\r\n

And this is the resulting mail headers of the mail on the receiving SMTP server:

Return-Path: <www-data@f3de6efeb3a0>
X-Envelope-To: [email protected]
Message-ID: <[email protected]>
Received: from localhost ([127.0.0.1])
	by mail.example.com with SMTP
	for [email protected];
	Sat, 12 Feb 2022 06:28:10 +0100
Received: by f3de6efeb3a0 (sSMTP sendmail emulation); Sat, 12 Feb 2022 06:28:10 +0100
From: "Linux User" <www-data@f3de6efeb3a0>
Date: Sat, 12 Feb 2022 06:28:10 +0100
To: [email protected]

This is the output using the exact same environment, just that the container is build using "FROM php:7.4-fpm-alpine":

Received: by ae23725475da (sSMTP sendmail emulation); Sat, 12 Feb 2022 06:27:33 +0100\r\n
From: "Linux User" <www-data@ae23725475da>\r\n
Date: Sat, 12 Feb 2022 06:27:33 +0100\r\n
To: [email protected]\r\n
Subject: This is the subject\r\n
\r\n
This is the Body\r\n

And this is the resulting mail headers of the mail on the receiving SMTP server:

Return-Path: <www-data@ae23725475da>
X-Envelope-To: [email protected]
Message-ID: <[email protected]>
Received: from localhost ([127.0.0.1])
	by mail.example.com with SMTP
	for [email protected];
	Sat, 12 Feb 2022 06:27:33 +0100
Received: by ae23725475da (sSMTP sendmail emulation); Sat, 12 Feb 2022 06:27:33 +0100
From: "Linux User" <www-data@ae23725475da>
Date: Sat, 12 Feb 2022 06:27:33 +0100
To: [email protected]
Subject: This is the subject

The result is that when using PHP 8 there are some invalid "\r" which makes the mail format invalid and so the headers are not processed correctly on the receiving SMTP server. The result is that the email does not have a subject. I've also tried setting additional headers, but they are all invalid and not processed due to the invalid "\r".
I've tried using both sendmail and ssmtp in the container. It yields the same result.

My php.ini setting are:

sendmail_path = '/usr/sbin/sendmail -t -i -S myserver:25'

Something has changed in the PHP mail() function between PHP 7 and PHP 8

PHP Version

PHP 8.0.14 and newer

Operating System

Docker php:8.0.14-fpm-alpine and php:fpm-alpine

[cmb69]

mail() now outputs proper email line endings (i.e. CRLF instead of LF). It looks like your MTAs convert that to CRCRLF, what would be a bug in those MTAs.

[hansherlighed]

mail() now outputs proper email line endings (i.e. CRLF instead of LF). It looks like your MTAs convert that to CRCRLF, what would be a bug in those MTAs.

The capture from tcpdump is taken on the PHP server, so the trace is captured when going FROM PHP to the MTA(it's the output from sendmail). So the CRCRLF seems to comes from PHP, not the MTA(unless you refer to sendmail as the MTA).
If I read the specifications correctly, PHP should output CRLF directly after each line(which it does in PHP 7.4), but does not in PHP 8.

When you say MTA, do you mean sendmail? Do you mean that it is sendmail that is converting the input from PHP to CRCRLF?
Is there anyway to see the raw output sent from PHP to sendmail?

[cmb69]

You can see the raw PHP output by e.g. writing to a file instead of using sendmail. I.e. set sendmail_path to something like tee mailBasic.out >/dev/null.

[hansherlighed]

Thanks @cmb69

I was able to get the raw output from the mail() function into a file using your advice.
You are indeed right, the raw output from PHP 8 looks like this:

cat -e raw_mailoutput_php8.out

To: [email protected]^M$
Subject: This is the subject^M$
^M$
This is the Body^M$

and in PHP 7 it looks like this:

cat -e raw_mailoutput_php7.out

To: [email protected]$
Subject: This is the subject$
$
This is the Body$

This issue has actually been reported to PHP way back in 2002:
https://bugs.php.net/bug.php?id=15841
The problem remains the same as reported in this bug. on UNIX systems, the line ending is expected to be "\n" as it is the native line ending for the system. Since PHP's mail() function on UNIX systems, is basically sending raw text output to a UNIX command(sendmail if you follow the documentation) the mail() function will not work as sendmail expects the line endings to be "\n" and NOT "\r\n".
By making this new change to the mail() function so it always uses "\r\n" for the build in headers for Subject: and To:, you've actually made things worse, and made your documentation invalid. It states in the mail() documentation page:

Note:

If messages are not received, try using a LF (\n) only. Some Unix mail transfer agents (most notably [» qmail](http:https://cr.yp.to/qmail.html)) replace LF by CRLF automatically (which leads to doubling CR if CRLF is used). This should be a last resort, as it does not comply with [» RFC 2822](http:https://www.faqs.org/rfcs/rfc2822).

Following this note will cause the output from PHP8 to be a mix of "\r\n" and "\n" for the headers. The build in headers for "To:" and "Subject:" will still use "\r\n" even if you set the additional headers to use only "\n".
running mail([email protected], "This is the subject", "This is the Body", "From: [email protected]\nContent-Type: text/plain"); results in:

cat -e raw_mailoutput_php8.out

To: [email protected]^M$
Subject: This is the subject^M$
From: [email protected]$
Content-Type: text/plain^M$
^M$
This is the Body^M$

The essence of the problem is that a UNIX command/program expects line endings to be "\n" and not "\r\n", and since the mail() function on UNIX systems, forwards/outputs to a UNIX command rather than talking to an SMTP server directly, it causes problems.
Why can't the mail() function use the same line endings as PHP_EOL, so that on windows it becomes "\r\n" and unix "\n"? That would make the function more platform independent.

The solution when using the "php:fpm-alpine" image is to set "sendmail_path" to:
'/usr/bin/dos2unix -u|/usr/sbin/sendmail -t -i'

[cmb69]

The essence of the problem is that a UNIX command/program expects line endings to be "\n" and not "\r\n", […]

That is irrelevant. The respective RFCs are Internet standards, and those mandate CRLF as line break, so MTAs should be able to deal with that regardless of the operating system they're running on. Consider to report this issue to the sendmail maintainers (if it still hasn't been fixed in the latest version).

[csandanov]

I stumbled upon the same issue recently.

The solution with dos2unix by @hansherlighed fixed it for me, thank you!

This issue happens both with busybox's and ssmtp's sendmail except with busybox's (unless you add verbosity) you will get this weird error sendmail: . failed when in fact it's the same RFC 2822 compliance error.

I don't know which SMTP server the OP used but I used OpenSMTPD, so possibly related OpenSMTPD/OpenSMTPD#1135

[speller]

I just faced this issue in the PHP8.1 alpine image. It was fine in PHP7 images.

[speller]

Consider to report this issue to the sendmail maintainers

Why PHP contributors didn't do this knowing that alpine builds are failing one of the basic functions?

[cmb69]

Why PHP contributors didn't do this knowing that alpine builds are failing one of the basic functions?

Because it is not a PHP issue. mail() creates mails with CRLF line endings according to the standard; MTAs need to deal with this.

[pfletch101]

I have just been bitten by this on a site hosted by a commercial ISP, where I don't have root access, don't know precisely what distro is being used, and dos2unix is not installed, so I can't use the workaround. I'm afraid that I don't agree that this isn't a PHP issue. Previous PHP versions implemented mail() by communicating directly with the MTA. In doing so, they presumably correctly followed the RFCs requiring "\r\n" as the end of line characters. For your own reasons (and I suspect that they were quite good ones), you have now decided to use sendmail as an intermediary. sendmail is designed and intended to work at and from the Unix command line. It is therefore entirely reasonable for sendmail to expect header and other 'lines' to be terminated by the standard Unix "\n". If PHP chooses to use sendmail as its intermediary in sending mail, it behoves PHP to comply with its formatting rules.

[cmb69]

PHP uses a configurable MTA (e.g. sendmail) on non Windows platforms for ages; nothing has changed in this regard. What has changed, is that as of PHP 8.0.0, the mail() function produces standard conforming header line endings throughout (i.e. CRLF, instead of LF). The first standard RFC which specifies these line endings is RFC 822 from August 13, 1982. If there are any MTAs which still cannot deal with that standard, I'd blame them. And whether these MTAs run on Linux or any other platform is irrelevant; the use of CRLF is mandated for all platforms.

dos2unix is not installed, so I can't use the workaround

You don't even need dos2unix for a workaround; something like sed /\r\n/\n/g should work as well, if you're allowed to change the sendmail_path setting.

[speller]

@pfletch101 Nobody cares. PHP team has implemented a standard breaking the whole functionality and just saying "That's not our business to solve this". The Docker team who build the Docker image says "consider installing an alternative image" and closed the ticket. The sendmail bugtracker is dead and nobody responded at all: mirror/busybox#58. In the end, we have a broken functionality, and everyone just ping-ponging users.

[michaelbrunnbauer]

Stuff is not useless just because it is old.

Right. So why don't you just stick with PHP 7.4?

You have no idea. I actually lost customers who would move somewhere else because they can keep using PHP 7.4 there. Most of the others had to be forced to upgrade their stuff - which was not easy for many of them. I know people who still use PHP 5.3. Screaming all day to them what a security risk that is doesn't change a thing. You have no idea. No idea at all.

[bukka]

Not all sendmail variants have the -ba option, e.g. qmail, netqmail, s/qmail.

According to Wikipedia, qmail had it's final release 24 years ago; netqmail 15 years ago. I would presume that such mailers have other issues, which might render them irrelevant. s/qmail seems to be maintained, though. However, are these even affected by this CRLF issue? Or do they perhaps have another option to conform to the 40 year old specs?

They are all affected and they are all alive. And there are many more variants of sendmail I am not familiar with. Stuff is not useless just because it is old. One does not always need newer features - especially on systems relaying mail to a smarthost. But please go ahead to the future and leave the past behind. I'll be smiling while the past catches up with the maintainers many many times in the next years.

Well we sometime need to stop supporting of some old versions and clients because it will become very difficult to make sure that the stuff still works there. In general we tend to do that when the support ends on the distributions that are still supported. For example currently the oldest distribution that we still support is RHEL 7 and its variants that are still alive (CentOS 7, Amazon Linux 2). That's how we decide to drop specific versions of the libraries.

This particular case is not such a case as it impacts many still supported emulations. However this problem is also causing some issue for other MTA as described in https://bugs.php.net/bug.php?id=47983 and PHP 8.0 was a good time to fix it. It was however omission to not keep the compatibility optionally which we should do if such change is done. However it's difficult to know that when making the change because it is very hard to actively test all MTA's. The only thing we can do is to add that option to the future version and learn from this that doing changes to mail output should require more testing and consider more MTA's.

[bukka]

I just created a PR #10191 that introduces an option to revert back to the previous behavior.

[michaelbrunnbauer]

BTW why are people pointing to rfc822? The introduction seems very clear to me:

Some message systems may store messages in formats that differ from the one specified in this standard. This specification is intended strictly as a definition of what message content format is to be passed BETWEEN hosts.

Note: This standard is NOT intended to dictate the internal formats used by sites, the specific message system features that they are expected to support, or any of the characteristics of user interface programs that create or read messages.

[bukka]

BTW why are people pointing to rfc822? The introduction seems very clear to me:

Some message systems may store messages in formats that differ from the one specified in this standard. This specification is intended strictly as a definition of what message content format is to be passed BETWEEN hosts.
Note: This standard is NOT intended to dictate the internal formats used by sites, the specific message system features that they are expected to support, or any of the characteristics of user interface programs that create or read messages.

I see the point here as technically it is up to MTA to follow the RFC when sending the message. This is really just a program interface. On the other side, the current format had its issue so an attempt to fix them was probably in place. I would personally treat this as an omission but I'm not sure if requiring compliance was in place here. I have to say that after doing some research I haven't found anywhere any spec for sendmail interface that would mandate message format to follow the RFC 822, 2822 or 5322. Honestly I found very little info in sendmail docs but I might have missed something so if anyone else sees it somewhere, please share your findings.

[speller]

@bukka Thank you a lot for the fix!

[ElliotNB]

@bukka Thank you for the fix! I was also recently bitten by this issue. Do you know when your GH-8086 fix will make it into the next PHP 8.1.* release? I was inspecting the most recent commits contained in 8.1.15 and this fix doesn't appear to be included: PHP-8.1.13...php-8.1.15

[bukka]

@ElliotNB The fix is not in 8.1 as it's somewhere between feature and a bug and it adds a new ini which is cleaner to do in the last supported version. Also there's a work around so I would suggest to use that in 8.0 and 8.1:

The solution when using the "php:fpm-alpine" image is to set "sendmail_path" to:
'/usr/bin/dos2unix -u|/usr/sbin/sendmail -t -i'

[aly9]

Dears, Is there any final class for sending emails to be used in PHP version greater than 8 without changing the sendmail_path ? Since we are not allowed to change the .ini file configuration in some shared hosting.

[bukka]

@aly9 no but you should contact the hosting provider to change it globally in their configs.

[djixas]

I'm experiencing this issue too and it's driving me insane , both on PHP 8,1 and 8,0, and i am not using alpine

Shardj/zf1-future#274

[aly9]

@bukka Thanks for your reply, but I have modify my sending emails script to use PHPMailer Class and it is working fine.

[zaphod77]

Sendmail binaries on linux, and pretty much every other OS using LF only are designed to parse normal text for the operating system when piped to directly.

They internally convert to match the RFC after, which causes a doubled up CR usually with php 8.x now.

Actual Sendmail does have a -ba parameter, which does place it into ARPANET mode, which will do the right thing when presented with a text file containing CRLFs for injection. specifying the -ba option in sendmail path does fix the issue, but naturally, the email tester doesn't do this. So the email test fails.

qmail's fake sendmail binary does NOT SUPPORT -ba. not even the alternate usendmail wrapper does. postfix, and most other MTAs that provide a fake sendmail binary do not support ARPANET mode like a real sendmail binary does. The only one that i've seen that does this is actual sendmail itself.

The sendmail binary has always accepted email for injection in the OS native text format. All other MTAs that have fake sendmail binaries do the same. They all still send RFC compliant emails, doing the needed conversion. When telnetting or s_clienting to the smtp port, telnet itself usually handles this part, and will convert the line endings properly.

I'm sorry, but PHP is wrong here. it's mail() function expects either smtp on windows, or a sendmail binary on linux/bsd/osx/etc. And sendmail has ALWAYS accepted messages in the UNIX (linefeed only) text scheme on unix, and it always will.

This isn't a silly RFC issue. Sendmail is NOT breaking the RFC by accepting and interpreting piped data stored in native OS text format. the RFC is for message transmission, and when using sendmail binary, php is not transmitting itself. it's handing off to the sendmail binary.

Exim is one that's smart enough to just do the right thing. It actually just strips out the \rs and then puts them back when it send the message :). I'm sure there are some others, but as we can see, it's just not widespread enough.

Now this IS something that configure should be able to catch.

actual sendmail will respond to
"sendmail -ba"
with
550 Recipient names must be specified

in this case, it's real sendmail, and -ba should be added to the options automatically.

Any other response mean sit's not actual sendmail, we can't do that.

[derStephan]

I think, PHP is mixing up responsibilities. There are 2 involved parties and both of them have their own tasks.

It is PHP's responsibility to assemble all information needed by the mail agent. And it is PHP's responsibility to hand that information over to the mail agent in way that is valid for the OS and the mail agent itself.

This is where PHP's responsibilities end.

The mail agent is responsible to take the information and create a Mail that complies to the respective RFCs.

We now have the situation that the standard way of sending mails is broken due to PHP's interference in something that is not their business.

I tried the new config variable mail.mixed_lf_and_crlf = On and at first this seemed to work as before. All mails look nice in Outlook an Thunderbird and everywhere. But I keep getting bounces from remote mail servers along the lines of

SMTP error from remote server for TEXT command, host: **.**.com (***.***.***.***) reason: 554 message body contains illegal bare CR/LF characters.

This is unacceptable. The only thing that seemed to work is undoing PHP's magic by setting

sendmail_path='/usr/bin/dos2unix -u|/usr/sbin/sendmail -t -i'

With this config setting, mails look good in all clients and there are no bounces. Seems like my mail agent (nullmailer) cares about RFC compliance quite fine on it's own.

May I ask you to revert this broken feature? Ensuring mailing RFCs is not PHP's responsibility.

[bukka]

mail.mixed_lf_and_crlf = on should be an exact revert of the breaking change. If you see some differences between what was in PHP 7 and what mail.mixed_lf_and_crlf = on does, please open a new issue (bug) with all the info and I will take a look.

[zaphod77]

The CORRECT behavior is to produce standard text in the operating system's standard text format when piping to sendmail, and to use CR+LF when using SMTP, which is the circumstance where having it be CRLF matters. Funny enough this setting is used on windows, which already uses CRLF.

PHP has a responsibility to pipe stuff to sendmail binaries in the format the sendmail binary expects, which is ALWAYS the native text format used by the operating system, unless it's real sendmail in ARPANET mode.

[bukka]

If you want any change to happen, you need to create a new issue. Comments in closed issues are pretty much useless because it just get forgotten.

[csandanov]

I also had issues with mail.mixed_lf_and_crlf = On
couldn't make it work with msmtp, so I keep using /usr/bin/dos2unix