[BUG] (PHP8.x - mail()) Notify #1348

realizelol · 2023-06-03T20:56:57Z

Hi Limetech,

PHP8.x uses the correct line endings in mail's headers [ "\r\n" (CRLF) ] or maybe ssmtp awaits it.
Here's more info about that on php: php/php-src#8086

$line_break = (PHP_VERSION_ID < 80000) ? "\n" : "\r\n";:
This will check if php version is less than 8.0 if so use "\n" else use "\r\n".

$headers[] = "";":
Remove newline in headers. Instead add this to the first line in body:
$body[] = "";:
Was added because otherwise the $body[] "Subject:" could also being interpreted as header.

$body = implode("\n", $body);:
Implode newlines into one string to a variable, so it can be wrapped by the next command:
$body = wordwrap($body, 70);
$headers = implode($line_break, $headers);
Don't need to be a variable but there's the $line_break which is needed in PHP8/SSMTP on Unraid 6.12.

return mail($to, $subj, $body, $headers);:
Return only variables.

I actually using this method in these two scripts:
/usr/local/emhttp/webGui/scripts/notify
or (symlink)
/usr/local/emhttp/plugins/dynamix/scripts/notify
otherwise the malformed header may result in error 554 by mail provider (e.g. https://postmaster.gmx.net/en/case?c=hi&i=ip):

function generate_email($event, $subject, $description, $importance, $message, $recipients, $fqdnlink) {
  global $ssmtp;
  $line_break  = (PHP_VERSION_ID < 80000) ? "\n" : "\r\n";
  $rcpt        = $ssmtp['RcptTo'];
  if (!$recipients)
    $to        = implode(',', explode(' ', trim($rcpt)));
  else
    $to        = $recipients;
  if (empty($to)) return;
  $subj        = "{$ssmtp['Subject']}$subject";
  $headers     = [];
  $headers[]   = "MIME-Version: 1.0";
  $headers[]   = "X-Mailer: PHP/".phpversion();
  $headers[]   = "Content-Type: text/plain; charset=UTF-8";
  $headers[]   = "From: {$ssmtp['root']}";
  $headers[]   = "Reply-To: {$ssmtp['root']}";
  if (($importance == "warning" || $importance == "alert") && $ssmtp['SetEmailPriority']=="True") {
    $headers[] = "X-Priority: 1 (highest)";
    $headers[] = "X-Mms-Priority: High";
  }
  $body        = [];
  $body[]      = "";
  if (!empty($fqdnlink)) {
    $body[]    = "Link: $fqdnlink";
    $body[]    = "";
  }
  $body[]      = "Event: $event";
  $body[]      = "Subject: $subject";
  $body[]      = "Description: $description";
  $body[]      = "Importance: $importance";
  if (!empty($message)) {
    $body[]    = "";
    foreach (explode('\n', $message) as $line)
    $body[]    = $line;
  }
  $body[]      = "";
  $body        = implode("\n", $body);
  $body        = wordwrap($body, 70);
  $headers     = implode($line_break, $headers);
  return mail($to, $subj, $body, $headers);
}

Affected function in file between line 52 and 88:
https://github.com/limetech/webgui/blob/master/emhttp/plugins/dynamix/scripts/notify#L52-L88

While WebGUI actually seems to working fine. The CLI isn't working - just give it a try by e.g.:

/usr/local/emhttp/plugins/dynamix/scripts/notify -b -e "Test" -s "Test Subject" -i normal \
  -d "Hello this is the short message." \
  -m "This is the long message\nYes here is line two of the long message\n...long three."

best regards

realizelol

The text was updated successfully, but these errors were encountered:

realizelol · 2023-06-12T23:04:17Z

It would also be possible to add the "Message-ID"-Header to also improve mail for e.g. gmail sees it critical.

$nginx       = @parse_ini_file('/var/local/emhttp/nginx.ini') ?: [];
$headers[]   = 'Message-ID: <'.time().'-'.md5(uniqid(time())).'@'.$nginx['NGINX_LANMDNS'].'>';

or use the senders domain:

$headers[]   = 'Message-ID: <'.time().'-'.md5(uniqid(time())).'@'.array_pop(explode('@', $ssmtp['root'])).'>';

or something fake:

$headers[]   = 'Message-ID: <'.time().'-'.md5(uniqid(time())).'@'.mailings.myunraid.net>';

I don't know how critical a mail provider will see each version..
But definitively it should be a FQDN, so gethostname() will not work OOTB.

best regards

realizelol

ljm42 · 2023-06-13T16:26:09Z

Thanks for your suggestions

We can simplify this since any new code will only be run on PHP 8:

$line_break  = (PHP_VERSION_ID < 80000) ? "\n" : "\r\n";:

But I am unclear what we are trying to solve with these changes? When I run this exact command in 6.12.0-rc8, it sends the email just fine:

/usr/local/emhttp/plugins/dynamix/scripts/notify -b -e "Test" -s "Test Subject" -i normal \
 -d "Hello this is the short message." \
 -m "This is the long message\nYes here is line two of the long message\n...long three."

I am using smtp.gmail.com to send mail and it works fine. Also without the Message-ID header.

realizelol · 2023-06-20T19:26:14Z

I've just updated to stable Release 6.12, but the error still exist.

/usr/local/emhttp/plugins/dynamix/scripts/notify -b -e "Test" -s "Test Subject" -i normal -d "Hello this is the short message." -m "This is the long message\nYes here is line two of the long message\n...long three."
ssmtp: 554 For explanation visit https://postmaster.gmx.net/en/case?c=hi&i=ip

This error should be related to the incorrect end of lines for the headers part.
But php mail() is also saying that the EOLs for the message should also be end by "\r\n":

Each line should be separated with a CRLF (\r\n). Lines should not be larger than 70 characters.
https://www.php.net/manual/en/function.mail.php

Message-Id header is complained by rspamd:
https://github.com/rspamd/rspamd/blob/master/rules/regexp/headers.lua#L129

-- Detects missing Message-ID
local has_mid = 'header_exists(Message-Id)'
reconf['MISSING_MID'] = {
  re = '!header_exists(Message-Id)',
  score = 2.5,
  description = 'Message-ID header is missing',
  group = 'headers',
  mime_only = true,
}

A spam-score of 2.5 is a relatively big deal, as 6 (add_header) is the default spam score:
https://github.com/rspamd/rspamd/blob/master/conf/actions.conf#L18

After that I did a google research and it showed up that there are also some errors with gmail
(I've just read the topics not the threads):
https://www.google.com/search?q=%22Message-ID%22+missing

after some debug - on another os:

Maybe there's an issue with the actual php-cli package.

I've tried the same php file on a LinuxMint and I get more or modified lines:

unraid	mint
dkim=none;	dkim=pass header.d=mail.com header.s=s31663417 header.b=GwE68pAT;
none	DKIM-Signature: (...Added correctly to the header...)
none	X-UI-Sender-Class: (...Added correctly to the header - just added by the mail provider - not relevant - but seems to be needed for adding the DKIM header...)
none	Message-ID: [email protected]
none	X-Provags-ID: (...Added correctly to the header...)
none	UI-OutboundReport: (...Added correctly to the header...)
none	X-Spam-Flag: (...Added correctly to the header...)

Message-ID + X-Provags-ID + X-Spam-Flag + UI-OutboundReport (headers) will be added to the body message.

So normally the Message-ID header is added by the mailprovider, so it's not needed to be added by ourself.

If not adding "\n" in front of $message the message also will be seen as header. So if adding "Subject: ..." in the message, my mail provider shows this error 554. https://postmaster.gmx.net/en/case?c=hi&i=ip -> Because I am sending two Subject headers.
And the headers added by my mail provider are shown in body message part, so I think that the actual php version will not close the headers part correctly.

the tested phpfile:

<?php
$message = "Line 1\r\nLine 2\r\nSubject: Line 3\r\nLoremipsumdolorsitametconsetetursadipscingelitrseddiamnonumyeirmodtemporinviduntutlaboreetdoloremagnaaliquyameratseddiam";
$message = wordwrap($message, 70, "\r\n", true);

$subject = "Test 123";
$from    = "[email protected]";
$to      = "[email protected]";

$headers = array(
    'From' => '[email protected],
    'Reply-To' => '[email protected]',
    'X-Mailer' => 'PHP/' . phpversion()
);
mail($to, $subject, "\n".$message, $headers);
?>

Edit:
PHP version: 8.1.2-1ubuntu2.11

realizelol · 2023-06-22T14:41:41Z

I've tested on a clean testing environment (6.12.0-rc2) these versions php7.4.27 with libsodium 1.0.18:
https://slackware.uk/slackware/slackware64-15.0/slackware64/n/php-7.4.27-x86_64-1.txz
https://slackware.uk/slackware/slackware64-15.0/slackware64/l/libsodium-1.0.18-x86_64-3.txz

and the error is gone.

realizelol · 2023-06-24T12:46:35Z

I've tested in a slackware-15.0 environment and got the same error.

But I figured out that php mail will do some crazy stuff:

Adding an empty line to body:

[<-] 220 gmx.net (mrgmx005) Nemesis ESMTP Service ready
[->] EHLO tower
[<-] 250 STARTTLS
[->] STARTTLS
[<-] 220 OK
[->] EHLO tower
[<-] 250 SIZE 69920427
[->] AUTH LOGIN
[<-] 334 *someUUID*
[->] *someUUID*
[<-] 334 *someUUID*
[<-] 235 Authentication succeeded
[->] MAIL FROM:<[email protected]>
[<-] 250 Requested mail action okay, completed
[->] RCPT TO:<[email protected]>
[<-] 250 OK
[->] DATA
[<-] 354 Start mail input; end with <CRLF>.<CRLF>
[->] Received: by tower (sSMTP sendmail emulation); Sat, 24 Jun 2023 14:00:48 +0200
[->] Date: Sat, 24 Jun 2023 14:00:48 +0200
[->] To: [email protected]
[->] Subject: Unraid - Notification
[->] MIME-Version: 1.0
[->] Content-Type: text/plain; charset=UTF-8
[->] From: [email protected]
[->] Reply-To: [email protected]
[->] X-Mailer: PHP/8.2.7
[->] 
[->] 
[->] Event: Unraid Status
[->] Subject: Notification
[->] Description: No description
[->] Importance: normal
[->] .
[<-] 250 Requested mail action okay, completed: id=*someUUID*
[->] QUIT
[<-] 221 gmx.net Service closing transmission channel

Removing the empty line to body again:

[<-] 220 gmx.net (mrgmx005) Nemesis ESMTP Service ready
[->] EHLO tower
[<-] 250 STARTTLS
[->] STARTTLS
[<-] 220 OK
[->] EHLO tower
[<-] 250 SIZE 69920427
[->] AUTH LOGIN
[<-] 334 *someUUID*
[->] *someUUID*
[<-] 334 *someUUID*
[<-] 235 Authentication succeeded
[->] MAIL FROM:<[email protected]>
[<-] 250 Requested mail action okay, completed
[->] RCPT TO:<[email protected]>
[<-] 250 OK
[->] DATA
[<-] 354 Start mail input; end with <CRLF>.<CRLF>
[->] Received: by unraid (sSMTP sendmail emulation); Sat, 24 Jun 2023 14:01:19 +0200
[->] Date: Sat, 24 Jun 2023 14:01:19 +0200
[->] To: [email protected]
[->] Subject: Unraid - Notification
[->] MIME-Version: 1.0
[->] Content-Type: text/plain; charset=UTF-8
[->] From: [email protected]
[->] Reply-To: [email protected]
[->] X-Mailer: PHP/8.2.7
[->] 
[->] Event: Unraid Status
[->] Subject: Notification
[->] Description: No description
[->] Importance: normal
[->] 
[->] .
[<-] 554 For explanation visit https://postmaster.gmx.net/en/case?c=hi&i=ip
ssmtp: 554 For explanation visit https://postmaster.gmx.net/en/case?c=hi&i=ip

I think the extra empty line to the first row will break adding headers by the mail provider.
Removing this line will add an extra line to the end of the message body, which will break mail completely or may be unsterstood as extra long header (X-Mailer in example) and the "Subject: " in message body is even understood as header.
In mail specification it's not allowed to send the header twice.

Long headers field: https://datatracker.ietf.org/doc/html/rfc5322#section-2.2.3
Line length could be up to 78 not 70, sorry: https://datatracker.ietf.org/doc/html/rfc5322#section-2.1.1
but 70 is written by php: https://www.php.net/manual/en/function.mail.php#refsect1-function.mail-parameters

Removing the last char of body e.g. $body = substr($body, 0, -1); will just cut the last char of the body message,
so it's definitively a bug of php mail().

test.txt:

To: [email protected]
From: [email protected]
Reply-to: [email protected]
Subject: Hello Test

This is the message.

ssmtp -t -v < test.txt works fine on unraid + slackware-15.0.

Maybe gmail is not yet that restrictive as gmx actually is, but I think it should be too:
https://support.google.com/mail/thread/180100549/message-is-not-rfc-5322-compliant-duplicate-headers?hl=en

The test message will add "Subject: Notification" to the body text:
/usr/local/emhttp/plugins/dynamix/scripts/notify -b -t and should break your message, too.

So it's definitively not a bug in unraid / webgui, sorry.

But I will let this ticket open until I got a working solution or bug fix by the php maintainer.

realizelol · 2023-06-24T15:06:35Z

SSMTP isn't understanding the new output with CRLF of mail() since php8.0.

piping the mail() output through sed (remove \r - CR LineEndings) will work:
sendmail_path = "sed \"s/\r//g\" | ssmtp -t"

Or just use the new php option to the php.ini:

; Use mixed LF and CRLF line separators to keep compatibility with some
; RFC 2822 non conformant MTA. (For ssmtp MTA use "On")
mail.mixed_lf_and_crlf = On

if ssmtp ever fix their behaviour we could remove the line.

Source:
@cc931af

But also remove #L71:
$headers[] = "";

and line #L86:
$body[] = "";

These empty lines aren't needed. But $body[] = ""; could be kept in case of better scrolling compatibility on mobile devices. The headers row break to the message body will be done by mail() function.

ljm42 · 2023-06-24T21:48:57Z

Thanks for reporting this. It works fine with gmail, but I was able to reproduce the issue using smtp.mail.com. Working on a fix.

ljm42 · 2023-06-25T17:50:37Z

In your testing, is this enough to solve the problem?
mail.mixed_lf_and_crlf = On

Or do we need to modify the code to remove the blank line from headers?
$headers[] = "";
For me it works fine without any code changes.

If code changes are needed, I'm thinking we should go all the way and pass an associative array of headers so that PHP handle all the line endings internally. But I'd rather hold that larger change for the next major release so it can get wider testing before being deployed.

realizelol · 2023-06-25T18:50:30Z

In my testing adding the line mail.mixed_lf_and_crlf = On to php.ini works fine.
Adding an empty line to the headers is understood as an extended header line to the last header which would be "Reply-To" or "X-Mms-Priority" in urgency (@ notify file). So the empty lines whether to headers nor to body isn't needed.

Also it's not necessary to edit the end of lines whether in headers nor in message body.
The line endings already changed by php-8.0.0RC2 @6983ae7
But this has broken some MTA's like ssmtp (If I remember correctly also qmail).
So since 8.2.3-RC2 there's php setting to output the old EOL "\n" instead the new style "\r\n" to mail() function.

As per specification the correct line ending for mail is "\r\n" CRLF. It's converted to CRLF by ssmtp - but it needs it's input in "\n" LF like some other MTA's, so we need the old line endings - like in php7.4 - style "\n".
This will be done by the boolean mail.mixed_lf_and_crlf:
On = "\n"
Off = "\r\n"

This are the corresponding lines to the php setting mail.mixed_lf_and_crlf @cc931af:

(..)
	char *line_sep = PG(mail_mixed_lf_and_crlf) ? "\n" : "\r\n";
(..)
		fprintf(sendmail, "To: %s%s", to, line_sep);
		fprintf(sendmail, "Subject: %s%s", subject, line_sep);
		if (hdr != NULL) {
			fprintf(sendmail, "%s%s", hdr, line_sep);
		}
		fprintf(sendmail, "%s%s%s", line_sep, message, line_sep);
(..)

I hope this will clarify your doubts otherwise feel free to ask me again. It even took me 3 weeks to debug and understand.

ljm42 · 2023-06-25T22:23:03Z

OK I think we are both saying the same thing :)

So php.ini in 6.12.2 will include this:
mail.mixed_lf_and_crlf = On
and that will resolve the issue we are seeing with sending email notifications to smtp servers such as smtp.mail.com. Note that smtp.gmail.com works with or without that change.

ljm42 · 2023-08-29T01:55:21Z

This was fixed in 6.12.2

ljm42 closed this as completed Aug 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[BUG] (PHP8.x - mail()) Notify #1348

[BUG] (PHP8.x - mail()) Notify #1348

realizelol commented Jun 3, 2023

realizelol commented Jun 12, 2023

ljm42 commented Jun 13, 2023

realizelol commented Jun 20, 2023 •

edited

Loading

realizelol commented Jun 22, 2023

realizelol commented Jun 24, 2023

realizelol commented Jun 24, 2023

ljm42 commented Jun 24, 2023

ljm42 commented Jun 25, 2023

realizelol commented Jun 25, 2023

ljm42 commented Jun 25, 2023

ljm42 commented Aug 29, 2023

[BUG] (PHP8.x - mail()) Notify #1348

[BUG] (PHP8.x - mail()) Notify #1348

Comments

realizelol commented Jun 3, 2023

realizelol commented Jun 12, 2023

ljm42 commented Jun 13, 2023

realizelol commented Jun 20, 2023 • edited Loading

after some debug - on another os:

realizelol commented Jun 22, 2023

realizelol commented Jun 24, 2023

realizelol commented Jun 24, 2023

ljm42 commented Jun 24, 2023

ljm42 commented Jun 25, 2023

realizelol commented Jun 25, 2023

ljm42 commented Jun 25, 2023

ljm42 commented Aug 29, 2023

realizelol commented Jun 20, 2023 •

edited

Loading