toolchain/musl: update to 1.2.4 #12555

misaka36 · 2023-05-07T12:40:59Z

Release notes:

new features:

large dns record lookups via tcp fallback
new getaddrinfo EAI_NODATA result to distinguish NODATA/NxDomain
support for new RELR compressed format for relative relocations
sysconf keys for querying signal stack size requirements
real vfork on riscv64

performance:

mallocng no longer uses MADV_FREE (high performance cost, little gain)
vdso clock_gettime is supported once again on 32-bit arm

compatibility:

gethostbyname family now distinguishes NO_DATA from HOST_NOT_FOUND
res_send now works with caller-provided edns0 queries
arpa/nameser.h RR types list is now up-to-date
previously-missing POSIX confstr keys have been added
mntent interfaces now accept missing fields
alt signal stack, if any, is now used for internal signals
the LFS64 macros are no longer exposed without _LARGEFILE64_SOURCE
memmem (POSIX-future) is now exposed in default feature profile
pthread_atfork now admits calls from an application-provided malloc
debugger tracking of shared libraries now works on MIPS PIE binaries
sendmsg now supports up to SCM_MAX_FD fds in SCM_RIGHTS messages

bugs fixed:

gethostbyname[2]_r wrongly returned nonzero (error) on negative result
parallel v4/v6 address queries could fail on query id collisions
spurious getaddrinfo/AI_ADDRCONFIG failures due to errno clobbering
dns search domains ending in dot (including lone dot) broke lookups
ipv6 servers in resolv.conf broke lookups on systems with v6 disabled
systems with bindv6only failed to query both v4 and v6 nameservers
res_mkquery mishandled consecutive final dots in name
res_send could malfunction for very small answer buffer sizes
resolver dns backend accepted answers with wrong (A vs AAAA) RR type
getservbyport_r returned junk or ENOENT (vs ERANGE) on buffer size errors
dns result parsing of malformed responses could process uninitialized data
freopen didn't reset stream orientation (byte/wide) & encoding rule
fwprintf didn't print most fields on open_wmemstream FILEs
wide printf %lc ignored field width
wide printf erroneously processed %n after encoding errors
use of wide printf %9$ argument slot overflowed undersized buffer
swprintf malfunctioned on nul character in output
strverscmp ordered digit sequences vs nondigits incorrectly
timer_create/SIGEV_THREAD failure leaked the thread
semaphores were subject to missed-wake under certain usage patterns
several possible rare deadlocks with lock handling at thread exit
several possible rare deadlocks with aio and multithreaded fork
dynamic linker relro processing was broken on archs w/variable pagesize
async cancellation could run cancellation handlers in invalid context
pthread_detach was wrongly a cancellation point in rare race code path
use-after-close/double-close errors in mq_notify error paths
mq_notify event thread wrongly ran with signals unmasked
wcs{,n}cmp, wmemcmp returned wrong results when difference overflowed
accept4, pipe2, and dup3 handled unknown flags wrong in fallback cases
CPU_SETSIZE macro had wrong unit
select fallback for pre-time64 kernels truncated timeout (vs clamping)

arch-specific bugs fixed:

x32 new socketcalls took fallback path due to pointer sign extension
x32 wait4 didn't fill rusage structure (time64 regression)
x32 semtimedop mismatched timespec ABI with kernel (time64 regression)
sigaction signal mask was bogus on or1k, microblaze, mips, and riscv
powerpc-sf longjmp asm clobbered value argument
or1k poll function passed timeout to syscall in wrong form

Removed upstreamed:

800-mips_pie_debug.patch

Manually rebased:

600-nftw-support-common-gnu-extension.patch

xback · 2023-05-07T13:07:28Z

Did you do a full compile test?
I also made this patch last week and noticed a linux tool that had a build issue with this bump.

misaka36 · 2023-05-07T13:29:20Z

Yes. https://github.com/lovehackintosh/openwrt/actions/runs/4906138517
libbsd had a build issue. That’s why I marked it as a draft.

xback · 2023-05-07T16:55:04Z

Thanks, but I recall it was another tool.
I'll provide detailed info tomorrow when I'm in the office.

xback · 2023-05-08T07:52:04Z

Here it is:

selinux_restorecon.c: In function 'filespec_add':
selinux_restorecon.c:425:23: error: storage size of 'sb' isn't known
  425 |         struct stat64 sb;
      |                       ^~
selinux_restorecon.c:438:31: warning: implicit declaration of function 'lstat64'; did you mean 'lstat'? [-Wimplicit-function-declaration]
  438 |                         ret = lstat64(fl->file, &sb);
      |                               ^~~~~~~
      |                               lstat
make[5]: *** [Makefile:167: selinux_restorecon.o] Error 1
make[5]: Leaving directory '/home/koen/firmware/builds/generic_rb922/build_dir/target-mips_24kc_musl/libselinux-3.3/src'
make[4]: *** [Makefile:48: all] Error 1
make[4]: Leaving directory '/home/koen/firmware/builds/generic_rb922/build_dir/target-mips_24kc_musl/libselinux-3.3'
make[3]: *** [Makefile:161: /home/koen/firmware/builds/generic_rb922/build_dir/target-mips_24kc_musl/libselinux-3.3/.built] Error 2
make[3]: Leaving directory '/home/koen/firmware/builds/generic_rb922/package/libs/libselinux'
time: package/libs/libselinux/compile#0.16#0.01#0.16
    ERROR: package/libs/libselinux failed to build.
make[2]: *** [package/Makefile:116: package/libs/libselinux/compile] Error 1
make[2]: Leaving directory '/home/koen/firmware/builds/generic_rb922'
make[1]: *** [package/Makefile:110: /home/koen/firmware/builds/generic_rb922/staging_dir/target-mips_24kc_musl/stamp/.package_compile] Error 2
make[1]: Leaving directory '/home/koen/firmware/builds/generic_rb922'
make: *** [/home/koen/firmware/builds/generic_rb922/include/toplevel.mk:231: world] Error 2

robimarko · 2023-05-08T10:22:38Z

@xback

On the API level, the legacy "LFS64" ("large file support") interfaces, which were provided by macros remapping them to their standard names (#define stat64 stat and similar) have been deprecated and are no longer provided under the _GNU_SOURCE feature profile, only under explicit _LARGEFILE64_SOURCE. The latter will also be removed in a future version. Builds broken by this change can be fixed short-term by adding -D_LARGEFILE64_SOURCE to CFLAGS, but should be fixed to use the standard interfaces.

So, _LARGEFILE64_SOURCE needs to be defined or passed via CFLAGS which works for me.
robimarko@147f65b

PolynomialDivision · 2023-05-12T22:38:12Z

It would also be nice if someone could bump selinux to the new versions.

bittorf · 2023-05-13T07:53:49Z

please change version number in title:
it is NOT "update to 1.24" but "update to 1.2.4"
Thanks for your work!

misaka36 · 2023-05-13T08:48:30Z

@robimarko Thank you for your patch. I applied it.
@bittorf Changed it, thank you.

hauke · 2023-05-13T12:27:11Z

Is this problem already fixed upstream in selinux? If not id you report it?

The problem in the github CI was fixed in #12590, maybe it will only pass again after the x86_64 SDK was build by the build bots.

robimarko · 2023-05-13T16:35:07Z

@hauke LFS64 is still used upstream, however there is an issue from 2022 regarding it:
SELinuxProject/selinux#366

Ansuel · 2023-05-15T17:48:17Z

@lovehackintosh @robimarko still draft or the problem got fixed?

robimarko · 2023-05-15T17:52:50Z

Its fixed now, second commit in the PR fixes selinux

Release notes: new features: - large dns record lookups via tcp fallback - new getaddrinfo EAI_NODATA result to distinguish NODATA/NxDomain - support for new RELR compressed format for relative relocations - sysconf keys for querying signal stack size requirements - real vfork on riscv64 performance: - mallocng no longer uses MADV_FREE (high performance cost, little gain) - vdso clock_gettime is supported once again on 32-bit arm compatibility: - gethostbyname family now distinguishes NO_DATA from HOST_NOT_FOUND - res_send now works with caller-provided edns0 queries - arpa/nameser.h RR types list is now up-to-date - previously-missing POSIX confstr keys have been added - mntent interfaces now accept missing fields - alt signal stack, if any, is now used for internal signals - the LFS64 macros are no longer exposed without _LARGEFILE64_SOURCE - memmem (POSIX-future) is now exposed in default feature profile - pthread_atfork now admits calls from an application-provided malloc - debugger tracking of shared libraries now works on MIPS PIE binaries - sendmsg now supports up to SCM_MAX_FD fds in SCM_RIGHTS messages bugs fixed: - gethostbyname[2]_r wrongly returned nonzero (error) on negative result - parallel v4/v6 address queries could fail on query id collisions - spurious getaddrinfo/AI_ADDRCONFIG failures due to errno clobbering - dns search domains ending in dot (including lone dot) broke lookups - ipv6 servers in resolv.conf broke lookups on systems with v6 disabled - systems with bindv6only failed to query both v4 and v6 nameservers - res_mkquery mishandled consecutive final dots in name - res_send could malfunction for very small answer buffer sizes - resolver dns backend accepted answers with wrong (A vs AAAA) RR type - getservbyport_r returned junk or ENOENT (vs ERANGE) on buffer size errors - dns result parsing of malformed responses could process uninitialized data - freopen didn't reset stream orientation (byte/wide) & encoding rule - fwprintf didn't print most fields on open_wmemstream FILEs - wide printf %lc ignored field width - wide printf erroneously processed %n after encoding errors - use of wide printf %9$ argument slot overflowed undersized buffer - swprintf malfunctioned on nul character in output - strverscmp ordered digit sequences vs nondigits incorrectly - timer_create/SIGEV_THREAD failure leaked the thread - semaphores were subject to missed-wake under certain usage patterns - several possible rare deadlocks with lock handling at thread exit - several possible rare deadlocks with aio and multithreaded fork - dynamic linker relro processing was broken on archs w/variable pagesize - async cancellation could run cancellation handlers in invalid context - pthread_detach was wrongly a cancellation point in rare race code path - use-after-close/double-close errors in mq_notify error paths - mq_notify event thread wrongly ran with signals unmasked - wcs{,n}cmp, wmemcmp returned wrong results when difference overflowed - accept4, pipe2, and dup3 handled unknown flags wrong in fallback cases - CPU_SETSIZE macro had wrong unit - select fallback for pre-time64 kernels truncated timeout (vs clamping) arch-specific bugs fixed: - x32 new socketcalls took fallback path due to pointer sign extension - x32 wait4 didn't fill rusage structure (time64 regression) - x32 semtimedop mismatched timespec ABI with kernel (time64 regression) - sigaction signal mask was bogus on or1k, microblaze, mips, and riscv - powerpc-sf longjmp asm clobbered value argument - or1k poll function passed timeout to syscall in wrong form Removed upstreamed: - 800-mips_pie_debug.patch Manually rebased: - 600-nftw-support-common-gnu-extension.patch Signed-off-by: Linhui Liu <[email protected]>

musl 1.2.4 deprecated legacy "LFS64" ("large file support") interfaces so just having _GNU_SOURCE defined is not enough anymore. _LARGEFILE64_SOURCE has to be defined in the source, or CFLAGS can be used to pass -D_LARGEFILE64_SOURCE to allow to keep using LFS64 definitions. Signed-off-by: Robert Marko <[email protected]>

1715173329 · 2023-05-16T04:29:33Z

making all in lib/ext2fs
make[5]: Entering directory '/mnt/disk/master/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/e2fsprogs-1.47.0/lib/ext2fs'
	CC bb_compat.c
	CC inode_io.c
	CC write_bb_file.c
	CC dupfs.c
	CC imager.c
	CC ext2_err.c
	CC alloc.c
	CC alloc_sb.c
	CC alloc_stats.c
	CC alloc_tables.c
	CC atexit.c
	CC badblocks.c
	CC bb_inode.c
	CC bitmaps.c
	CC bitops.c
	CC blkmap64_ba.c
	CC blkmap64_rb.c
	CC blknum.c
	CC block.c
	CC bmap.c
	CC check_desc.c
	CC closefs.c
	CC crc16.c
	GEN32CTABLE crc32c_table.h
	CC crc32c.c
	CC llseek.c
llseek.c:70:22: error: expected declaration specifiers or '...' before '_llseek'
   70 | static _syscall5(int,_llseek,unsigned int,fd,unsigned long,offset_high,
      |                      ^~~~~~~
llseek.c:70:43: error: unknown type name 'fd'
   70 | static _syscall5(int,_llseek,unsigned int,fd,unsigned long,offset_high,
      |                                           ^~
llseek.c:70:60: error: unknown type name 'offset_high'
   70 | static _syscall5(int,_llseek,unsigned int,fd,unsigned long,offset_high,
      |                                                            ^~~~~~~~~~~
llseek.c:71:33: error: unknown type name 'offset_low'; did you mean 'offsetof'?
   71 |                  unsigned long, offset_low,ext2_loff_t *,result,
      |                                 ^~~~~~~~~~
      |                                 offsetof
llseek.c:71:58: error: unknown type name 'result'
   71 |                  unsigned long, offset_low,ext2_loff_t *,result,
      |                                                          ^~~~~~
llseek.c:72:32: error: unknown type name 'origin'
   72 |                  unsigned int, origin);
      |                                ^~~~~~
llseek.c:67:12: warning: '_llseek' used but never defined
   67 | static int _llseek (unsigned int, unsigned long,
      |            ^~~~~~~
make[5]: *** [Makefile:693: llseek.o] Error 1
make[5]: Leaving directory '/mnt/disk/master/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/e2fsprogs-1.47.0/lib/ext2fs'
make[4]: *** [Makefile:456: all-libs-recursive] Error 1
make[4]: Leaving directory '/mnt/disk/master/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/e2fsprogs-1.47.0'
make[3]: *** [Makefile:371: all] Error 2
make[3]: Leaving directory '/mnt/disk/master/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/e2fsprogs-1.47.0'
make[2]: *** [Makefile:319: /mnt/disk/master/build_dir/target-arm_cortex-a15+neon-vfpv4_musl_eabi/e2fsprogs-1.47.0/.built] Error 2
make[2]: Leaving directory '/mnt/disk/master/package/utils/e2fsprogs'
time: package/utils/e2fsprogs/compile#2.01#0.37#2.24
    ERROR: package/utils/e2fsprogs failed to build.
make[1]: *** [package/Makefile:120: package/utils/e2fsprogs/compile] Error 1
make[1]: Leaving directory '/mnt/disk/master'
make: *** [/mnt/disk/master/include/toplevel.mk:231: package/compile] Error 2

looks like e2fsprogs exploded due to this update

Edit: need to pass -D_LARGEFILE64_SOURCE flag to e2fsprogs and f2fs-tools

robimarko · 2023-05-16T07:56:21Z

Did you rebuild the toolchain?

Cause, e2fsprogs is building just fine for me without any changes at all.

1715173329 · 2023-05-16T07:57:58Z

Did you rebuild the toolchain?

yep, i did clean up everything.
alpine also added the flag https://git.alpinelinux.org/aports/commit/main/e2fsprogs/APKBUILD?id=1316b68e9bb502e94953f61c7e16fa73771b0b93

robimarko · 2023-05-16T07:59:03Z

Hm, its rather weird that its compiling for me, f2fs-tools do require LFS64 to be reenabled though.

1715173329 · 2023-05-16T08:03:14Z

Hm, its rather weird that its compiling for me

hmmmm looks like it only fails for 32 bit targets

hauke · 2023-05-16T08:07:11Z

libbsd is still not compiling:

 make[3] -C package/libs/libbsd compile
    ERROR: package/libs/libbsd failed to build.
make: *** [/__w/openwrt/openwrt/openwrt/include/toplevel.mk:231: world] Error 1
make -r world: build failed. Please re-run make with -j1 V=s or V=sc for a higher verbosity level to see what's going on
====== Make errors from logs/package/libs/libbsd/compile.txt ======
funopen.c:68:28: error: unknown type name 'off64_t'; did you mean 'off_t'?
   68 | funopen_seek(void *cookie, off64_t *offset, int whence)
      |                            ^~~~~~~
      |                            off_t
funopen.c: In function 'funopen':
funopen.c:111:25: error: 'funopen_seek' undeclared (first use in this function); did you mean 'funopen_read'?
  111 |                 .seek = funopen_seek,
      |                         ^~~~~~~~~~~~
      |                         funopen_read
funopen.c:111:25: note: each undeclared identifier is reported only once for each function it appears in
make[6]: *** [Makefile:663: funopen.lo] Error 1
make[6]: Leaving directory '/__w/openwrt/openwrt/openwrt/build_dir/target-arm_mpcore_musl_eabi/libbsd-0.11.7/src'
make[5]: *** [Makefile:415: all-recursive] Error 1
make[5]: Leaving directory '/__w/openwrt/openwrt/openwrt/build_dir/target-arm_mpcore_musl_eabi/libbsd-0.11.7'
make[4]: *** [Makefile:347: all] Error 2
make[4]: Leaving directory '/__w/openwrt/openwrt/openwrt/build_dir/target-arm_mpcore_musl_eabi/libbsd-0.11.7'
make[3]: *** [Makefile:49: /__w/openwrt/openwrt/openwrt/build_dir/target-arm_mpcore_musl_eabi/libbsd-0.11.7/.built] Error 2
time: package/libs/libbsd/compile#23.13#2.14#27.23
Error: Process completed with exit code 2.

robimarko · 2023-05-16T08:07:24Z

That would explain why I am not seeing it.
f2fs-tools have some kind of a clash with fortify-headers which makes no sense as f2fs-tools correctly use #define _LARGEFILE64_SOURCE

robimarko · 2023-05-16T08:11:13Z

@hauke It needs TARGET_CFLAGS += -D_LARGEFILE64_SOURCE as well.

1715173329 · 2023-05-16T09:52:34Z

f2fs-tools have some kind of a clash with fortify-headers which makes no sense as f2fs-tools correctly use #define _LARGEFILE64_SOURCE

from configuare script output:

checking for lseek64... no

https://git.kernel.org/pub/scm/linux/kernel/git/jaegeuk/f2fs-tools.git/tree/lib/libf2fs_io.c#n70
looks like it just failed to detect lseek64 function and redefined it

1715173329 · 2023-05-16T14:26:53Z

not sure how many packages are/will be exploded, this shouldn't be merged before branching off...

1715173329 · 2023-05-17T04:04:53Z

https://downloads.openwrt.org/snapshots/faillogs/aarch64_cortex-a53/packages/
looks awesome

github-actions bot added the toolchain pull request/issue with toolchain related changes label May 7, 2023

misaka36 force-pushed the musl branch from 6bcdf66 to 1839958 Compare May 13, 2023 08:45

misaka36 changed the title ~~toolchain/musl: update to 1.24~~ toolchain/musl: update to 1.2.4 May 13, 2023

github-actions bot added the core packages pull request/issue for core (in-tree) packages label May 13, 2023

Ansuel marked this pull request as ready for review May 15, 2023 17:54

misaka36 and others added 2 commits May 15, 2023 20:39

Ansuel force-pushed the musl branch from 1839958 to 6ff1209 Compare May 15, 2023 18:39

openwrt-bot merged commit 6ff1209 into openwrt:master May 15, 2023
5 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

toolchain/musl: update to 1.2.4 #12555

toolchain/musl: update to 1.2.4 #12555

misaka36 commented May 7, 2023

xback commented May 7, 2023

misaka36 commented May 7, 2023

xback commented May 7, 2023

xback commented May 8, 2023

robimarko commented May 8, 2023 •

edited

PolynomialDivision commented May 12, 2023

bittorf commented May 13, 2023

misaka36 commented May 13, 2023

hauke commented May 13, 2023 •

edited

robimarko commented May 13, 2023 •

edited

Ansuel commented May 15, 2023

robimarko commented May 15, 2023

1715173329 commented May 16, 2023 •

edited

robimarko commented May 16, 2023 •

edited

1715173329 commented May 16, 2023

robimarko commented May 16, 2023

1715173329 commented May 16, 2023 •

edited

hauke commented May 16, 2023

robimarko commented May 16, 2023

robimarko commented May 16, 2023 •

edited

1715173329 commented May 16, 2023

1715173329 commented May 16, 2023

1715173329 commented May 17, 2023

toolchain/musl: update to 1.2.4 #12555

toolchain/musl: update to 1.2.4 #12555

Conversation

misaka36 commented May 7, 2023

xback commented May 7, 2023

misaka36 commented May 7, 2023

xback commented May 7, 2023

xback commented May 8, 2023

robimarko commented May 8, 2023 • edited

PolynomialDivision commented May 12, 2023

bittorf commented May 13, 2023

misaka36 commented May 13, 2023

hauke commented May 13, 2023 • edited

robimarko commented May 13, 2023 • edited

Ansuel commented May 15, 2023

robimarko commented May 15, 2023

1715173329 commented May 16, 2023 • edited

robimarko commented May 16, 2023 • edited

1715173329 commented May 16, 2023

robimarko commented May 16, 2023

1715173329 commented May 16, 2023 • edited

hauke commented May 16, 2023

robimarko commented May 16, 2023

robimarko commented May 16, 2023 • edited

1715173329 commented May 16, 2023

1715173329 commented May 16, 2023

1715173329 commented May 17, 2023

robimarko commented May 8, 2023 •

edited

hauke commented May 13, 2023 •

edited

robimarko commented May 13, 2023 •

edited

1715173329 commented May 16, 2023 •

edited

robimarko commented May 16, 2023 •

edited

1715173329 commented May 16, 2023 •

edited

robimarko commented May 16, 2023 •

edited