open() with O_WRONLY tries to incorrectly access the mode variable.

[fjpanag]

As I see here, this is the standard prototype for the open() function:

int open(const char *path, int oflag, ... );

According to the standard, the varargs may be parsed as a mode_t variable, but only if O_CREAT or O_TMPFILE is specified in the file flags.

If neither of the two is specified, then the extra arguments are not needed, and are ignored.

From open group:

O_CREAT
If the file exists, this flag has no effect except as noted under O_EXCL below. Otherwise, the file is created; the user ID of the file is set to the effective user ID of the process; the group ID of the file is set to the group ID of the file's parent directory or to the effective group ID of the process; and the access permission bits (see <sys/stat.h>) of the file mode are set to the value of the third argument taken as type mode_t modified as follows: a bitwise-AND is performed on the file-mode bits and the corresponding bits in the complement of the process' file mode creation mask. Thus, all bits in the file mode whose corresponding bit in the file mode creation mask is set are cleared. When bits other than the file permission bits are set, the effect is unspecified. The third argument does not affect whether the file is open for reading, writing or for both.

From Linux:

The mode argument specifies the file mode bits to be
applied when a new file is created. If neither O_CREAT
nor O_TMPFILE is specified in flags, then mode is ignored
(and can thus be specified as 0, or simply omitted). The
mode argument must be supplied if O_CREAT or O_TMPFILE is
specified in flags; if it is not supplied, some arbitrary
bytes from the stack will be applied as the file mode.

---

However, this is not the case in NuttX.
As I see here, NuttX will try to parse ap also in the case of O_WRONLY.

As far as I understand, even O_WRONLY will create the file if it does not exist, but I cannot see this referenced in any of the standards.

This is a portability and standards conformance issue.
Code ported from other platforms may use O_WRONLY without specifing the mode. In this case open() will read garbage from the stack.

[fjpanag]

Due to the way that NuttX handles the open flags, also O_RDWR will read garbage from the stack.