add meson based buildsystem v2

[fischerling]

add a meson based build system

This is based on and supersedes the meson PR #297.

I am willing to maintain the meson build system and provide help for all those
not familiar with meson (@tp-m and @stalkerg are also willing to help).
Anybody can reach out to me using [email protected].

Additionally, I am already following the io_uring mailing list and will also respond
there.

@axboe Should I post this also to the io_uring mailing list?

TODOS

• support the no Glibc feature provided by the current build system
• test the meson build system with GitHub bot
• decide the minimum required meson version I settled with meson >=0.53

---

git request-pull output:

The following changes since commit 1842b2a74f4e914cb094019d0f339baeffa3023b:

  examples/io_uring-udp: Use a proper cast for `(struct sockaddr *)` argument (2022-07-26 10:46:47 -0600)

are available in the Git repository at:

  https://github.com/fischerling/liburing/ meson-liburing

for you to fetch changes up to e29fafb12c40fa5de37b16bb21872c4f2476075c:

  meson: update available examples to liburing 2.3 (2022-07-27 17:19:15 +0200)

----------------------------------------------------------------
Florian Fischer (8):
      meson: update meson build files for liburing 2.3
      meson: update available tests to liburing 2.3
      meson: update installed manpages to liburing 2.3
      meson: add default test setup running each test once
      meson: support building without libc
      meson: add 'raw' test suite
      github bot: add jobs for meson
      meson: update available examples to liburing 2.3

Peter Eszlari (1):
      add Meson build system

 .github/workflows/build.yml      |  45 ++++++++++++--
 .gitignore                       |   2 +
 examples/meson.build             |  19 ++++++
 man/meson.build                  | 116 ++++++++++++++++++++++++++++++++++
 meson.build                      | 119 +++++++++++++++++++++++++++++++++++
 meson_options.txt                |  14 +++++
 src/include/liburing/compat.h.in |   7 +++
 src/include/liburing/meson.build |  51 +++++++++++++++
 src/include/meson.build          |   3 +
 src/meson.build                  |  28 +++++++++
 test/meson.build                 | 219 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 11 files changed, 619 insertions(+), 4 deletions(-)
 create mode 100644 examples/meson.build
 create mode 100644 man/meson.build
 create mode 100644 meson.build
 create mode 100644 meson_options.txt
 create mode 100644 src/include/liburing/compat.h.in
 create mode 100644 src/include/liburing/meson.build
 create mode 100644 src/include/meson.build
 create mode 100644 src/meson.build
 create mode 100644 test/meson.build

---

Click to show/hide pull request guidelines

Pull Request Guidelines

1. To make everyone easily filter pull request from the email
   notification, use [GIT PULL] as a prefix in your PR title.

[GIT PULL] Your Pull Request Title

2. Follow the commit message format rules below.
3. Follow the Linux kernel coding style (see: https://github.com/torvalds/linux/blob/master/Documentation/process/coding-style.rst).

Commit message format rules:

1. The first line is title (don't be more than 72 chars if possible).
2. Then an empty line.
3. Then a description (may be omitted for truly trivial changes).
4. Then an empty line again (if it has a description).
5. Then a Signed-off-by tag with your real name and email. For example:

Signed-off-by: Foo Bar <[email protected]>

The description should be word-wrapped at 72 chars. Some things should
not be word-wrapped. They may be some kind of quoted text - long
compiler error messages, oops reports, Link, etc. (things that have a
certain specific format).

Note that all of this goes in the commit message, not in the pull
request text. The pull request text should introduce what this pull
request does, and each commit message should explain the rationale for
why that particular change was made. The git tree is canonical source
of truth, not github.

Each patch should do one thing, and one thing only. If you find yourself
writing an explanation for why a patch is fixing multiple issues, that's
a good indication that the change should be split into separate patches.

If the commit is a fix for an issue, add a Fixes tag with the issue
URL.

Don't use GitHub anonymous email like this as the commit author:

[email protected]

Use a real email address!

Commit message example:

src/queue: don't flush SQ ring for new wait interface

If we have IORING_FEAT_EXT_ARG, then timeouts are done through the
syscall instead of by posting an internal timeout. This was done
to be both more efficient, but also to enable multi-threaded use
the wait side. If we touch the SQ state by flushing it, that isn't
safe without synchronization.

Fixes: https://github.com/axboe/liburing/issues/402
Signed-off-by: Jens Axboe <[email protected]>

---

By submitting this pull request, I acknowledge that:

1. I have followed the above pull request guidelines.
2. I have the rights to submit this work under the same license.
3. I agree to a Developer Certificate of Origin (see https://developercertificate.org for more information).

[ammarfaizi2]

Commit 8ec4e25 ("add Meson build system") is missing a Signed-off-by tag from the committer. If you pick someone's else patch, you have to put your Signed-off-by tag as well.

[ammarfaizi2]

TODOS

• support the current no Glibc feature provided by the current build system
• improve meson test tooling for running the tests in a loop (make runtests-loop)
• replace the custom current build system
• document the meson build system and its maintainers
• provide a simple convenient make wrapper

Should we test the meson build system with GitHub bot too?
If yes, please add it to the TODO list please.

[axboe]

Two things from just a quick look:

1. We should not retain the old build system. If we're doing meson, we're going all the way
2. Exception to the above might be that if I type 'make', I'd like it to just build with meson. This is similar to what eg happens with qemu, and is nice for folks that just expect 'make' to work without needing to know how to build with meson.

[axboe]

provide a simple convenient make wrapper

Ah that was already in the TODO, excellent.

[axboe]

Should we test the meson build system with GitHub bot too?

Most certainly.

[fischerling]

I am thinking about the minimum required meson version.

meson >= 0.60 would allow implementing the current make install-tests
make target using meson installation tags.

Currently we require meson >= 0.57 to ensure support for add_test_setup used
to provide convenient default meson test behavior.
Otherwise the test definitions use the fs meson module requiring at least meson >= 0.54.
Both current requirements can easily be dropped to support older meson versions.

Debian Bullseye for example ships meson 0.56.

Should we consider meson versions included by distributions at all?
I consider being compatible with old meson versions not that important since the
meson build system is not in the recent liburing-2.2 release and meson, being a
standalone python app, is really easy to install without a distro's package manager.

[eli-schwartz]

Currently we require meson >= 0.57 to ensure support for add_test_setup used
to provide convenient default meson test behavior.

It's possible to degrade this on older versions of Meson, by guarding the add_test_setup() in a meson.version() .version_compare(), such that the test setup is only defined on new enough versions of Meson.

On the plus side, this means that you can still configure the project on 0.56.0 using Debian Bullseye's distro-provided Meson. On the negative side, this means that if you try running meson test on such old versions, you don't get that convenient default behavior, but instead run the loop and parallel tests too, which is, uh, not wanted. :D

A hybrid approach could work, if backwards-compatibility is needed. Only with new Meson, define the loop/parallel tests and register the default test setup. Anyone who not only wants to run the tests, but also wants to run them in a loop... will simply be required to run a slightly more modern version of Meson (it's compatible with every single version of python that isn't end of life).

[axboe]

We should not retain the old build system. If we're doing meson, we're going all the way

After pondering this some more, I do believe we should keep the original as well. The point of adding meson should be easier integration for folks. Building liburing is already mostly as trivial as it can be.

That does have the downside of then having two build systems, but I don't think this will be too bad as it's actually pretty simple. It does come with the cost of making sure we stay in sync, we'll just have to see how that goes over time.

But if we retain both, perhaps that can simplify the meson side? We don't really need runtests support, for example, building and installing the core + man pages would really be all that's required. No?

[stalkerg]

I do believe we should keep the original as well.

From my point of view, it's a good strategy for a while to look at how Meson will work and how folks will adopt it before completely dropping Makefile.

But if we retain both, perhaps that can simplify the meson side? We don't really need runtests support, for example, building and installing the core + man pages would really be all that's required. No?

not really, it's not so complicated and I believe both build systems should be full functionally.

[stalkerg]

improve meson test tooling for running the tests in a loop

do we have an issue on the meson tracker? I really want to avoid any shell script run from Meson.

[eli-schwartz]

A shell script is needed in order to implement custom functionality above and beyond merely executing the compiled test executables. Particularly, grepping dmesg for possible errors. So I think shell scripts are unavoidable (and honestly not a real problem IMO).

[tp-m]

If a test should fail if there's something in the dmesg then perhaps that should be incorporated into the test or runner itself so that it returns a negative return value then? That way you can just run meson test --repeat=1000 and have it work automatically. (This is probably out of scope for this PR, just thinking out loud)

[fischerling]

improve meson test tooling for running the tests in a loop

do we have an issue on the meson tracker? I really want to avoid any shell script run from Meson.

The runtests-loop make target executes all tests in an infinite loop exiting only if a test failed.
Not sure how we would port such behavior to the meson test runner.

I see two possible solutions:

1. Add a meson test argument should_timeout similar to should_fail specifying that we expect the loop to not exit due to a test failure. @stalkberg do you think I should open an issue on the meson tracker?
2. Do not implement the looping tests with meson at all. We could simply call the runtests_loop.sh helper from the top-level Makefile.

I think I prefer the second variant.
It is easier to implement and does not change the behavior from the status quo.

[eli-schwartz]

If a test should fail if there's something in the dmesg then perhaps that should be incorporated into the test or runner itself so that it returns a negative return value then?

This is already the case, that's what the shell script wrapper does.

That way you can just run meson test --repeat=1000 and have it work automatically.

Hmm, good point, this may already work okay like that. Really high numbers is not quite the same as infinite, but for all intents and purposes...

[stalkerg]

../src/register.c:335:54: error: unused parameter 'flags' [-Werror,-Wunused-parameter]
                               struct io_uring_buf_reg *reg, unsigned int flags)

we should reduce the strictness of checking or changing API (as I understand we are not interesting in)
probably @axboe should look into

[axboe]

Yes, that is deliberately left unused for now, it's so we can avoid adding Yet Another identical function when we do add flags for this.

[axboe]

I'd say keep the unused variables, but add annotations so we can do __maybe_unused to silence them. Then when they do get used, we can just remove that annotation.

[axboe]

Done

[stalkerg]

@fischerling can you rebase PR or merge with master to pull updates.

[stalkerg]

Build test / build (x86_64, clang, clang, clang, clang++)

still failed

[fischerling]

Build test / build (x86_64, clang, clang, clang, clang++)

still failed

Sorry about that. The unnecessary explicit setting of the FLAGS in the workflow overrode the -Wno-unused-parameter and -Wno-sign-compare flags.

[ammarfaizi2]

From https://github.com/fischerling/liburing/commit/111dc30d980abe0126d513f9839f56aab16c258e.patch:

From 111dc30d980abe0126d513f9839f56aab16c258e Mon Sep 17 00:00:00 2001
From: Florian Fischer <[email protected]>
Date: Tue, 28 Jun 2022 10:28:13 +0200
Subject: [PATCH] remove obsolte Makefiles

Delete the custom build system superseded by meson.

Signed-off-by: Florian Fischer <[email protected]>
---
 .gitignore        |  19 --
 Makefile          |   2 -
 Makefile.common   |   6 -
 Makefile.quiet    |  11 --
 configure         | 467 ----------------------------------------------
 examples/Makefile |  38 ----
 liburing.pc.in    |  12 --
 src/Makefile      |  89 ---------
 test/Makefile     | 247 ------------------------
 9 files changed, 891 deletions(-)
 delete mode 100644 Makefile.common
 delete mode 100644 Makefile.quiet
 delete mode 100755 configure
 delete mode 100644 examples/Makefile
 delete mode 100644 liburing.pc.in
 delete mode 100644 src/Makefile
 delete mode 100644 test/Makefile

No, please keep the original build script and Makefile. Jens has said
so too ( #622 (comment) ).

I think we should make meson an optional thing rather than a
requirement for the build.

[eli-schwartz]

Regardless of any other factor, it's usually prudent to implement build system migration as a multi-stage process.

First implement the new build system, then let people try it out and decide whether it suits all their needs, potentially wait a release cycle to give people a chance to shake out any bugs.

And only then decide to either:

• maintain two build systems if desired and people find advantages in each one (the maintenance cost should be minor, it's not that complex of a build)
• declare that the new build system has achieved victory by meeting all the target requirements (including "no one still prefers the other build system"), and it's now safe to remove the old build system

It's surprisingly common for projects to support two build systems, anyway.

[kloczek]

Is it possible to updated/rediff that PR against latest version

+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
1 out of 1 hunk FAILED -- saving rejects to file configure.rej
1 out of 1 hunk FAILED -- saving rejects to file src/Makefile.rej
1 out of 1 hunk FAILED -- saving rejects to file test/Makefile.rej

[kloczek]

Just in case .. I'm not compalinig.
I'm just messanger 😋

[stalkerg]

ok, what next?

[ammarfaizi2]

[ ] decide the minimum required meson version

Any update on this TODO?
Is this important?

[fischerling]

[ ] decide the minimum required meson version

Any update on this TODO? Is this important?

I have no strong opinion on this.

Since some distros still ship meson 0.53 we could try to make all features
requiring a later meson version optional.

But since getting from >= 0.54, required by our use of the meson filesystem module,
to >= 0.53 requires rewriting our test definitions, and the current build system is
supported and usable, I would keep our current requirement of meson >= 0.57.

[ammarfaizi2]

[ ] decide the minimum required meson version

Any update on this TODO? Is this important?

I have no strong opinion on this.

Since some distros still ship meson 0.53 we could try to make all features
requiring a later meson version optional.

But since getting from >= 0.54, required by our use of the meson filesystem module,
to >= 0.53 requires rewriting our test definitions, and the current build system is
supported and usable, I would keep our current requirement of meson >= 0.57.

@eli-schwartz any comment?

[eli-schwartz]

Debian packages Meson 0.56.2, and Ubuntu 20.04 packages Meson 0.53.2

Sticking to 0.53 here, means the build is compatible with both. Sticking to 0.54 here, means that the build is compatible with Debian out of the box (but not with Ubuntu).

We could avoid rewriting all the test definitions by sticking with 0.54 but not going all the way to 0.57

...

Any system that has 0.53 already, can install 0.57 using pip install fairly easily, as Meson tries to be close to zero-dependency -- the only dependency is the "python" program itself. And the minimum version of python for both Meson 0.53 and Meson 0.57 is Python 3.6.

Of course, it's very convenient to be able to build with the distro version. On the other hand, the latest LTS of Ubuntu has a perfectly good enough version of Meson -- it's only old versions of Ubuntu that are stuck on 0.53, and if you're going to let that stand in the way of progress you cannot use new things for the better part of a decade...

[fischerling]

So I decided on meson >=0.53 to support more distros without the need of manual meson installation.

I dropped the use of the meson filesystem module from test/meson.build and use split('.') on the file name instead. This has only the limitation that test file names are not allowed to contain dots, which I think is a reasonable limitation.

The add_test_setup is wrapped in a meson version check as suggested by @eli-schwartz.

[fischerling]

@ammarfaizi2 I have slightly changed the github bot config (Install meson from ubuntu repos, do not pass compilers to nolibc build via environment).
Would you please check the config again. I would like to include your ack header when posting this pacth set to the mailinglist.

[ammarfaizi2]

@ammarfaizi2 I have slightly changed the github bot config (Install meson from ubuntu repos, do not pass compilers to nolibc build via environment).
Would you please check the config again. I would like to include your ack header when posting this pacth set to the mailinglist.

Looks fine to me.

Acked-by: Ammar Faizi <[email protected]>

Thanks!

[kloczek]

Cannot cleanly apply this PR on top of https://github.com/axboe/liburing//archive/liburing-2.2.tar.gz

+ /usr/bin/cat /home/tkloczko/rpmbuild/SOURCES/liburing-add-meson-based-buildsystem-v2.patch
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
2 out of 5 hunks FAILED -- saving rejects to file .github/workflows/build.yml.rej

[kloczek]

After manually remove patching .github/workflows/build.yml meson fails with

+ /usr/bin/meson --buildtype=plain --prefix=/usr --libdir=/usr/lib64 --libexecdir=/usr/libexec --bindir=/usr/bin --sbindir=/usr/sbin --includedir=/usr/include --datadir=/usr/share --mandir=/usr/share/man --infodir=/usr/share/info --localedir=/usr/share/locale --sysconfdir=/etc --localstatedir=/var --sharedstatedir=/var/lib --wrap-mode=nodownload --auto-features=enabled . x86_64-redhat-linux-gnu
The Meson build system
Version: 0.63.2
Source dir: /home/tkloczko/rpmbuild/BUILD/liburing-liburing-2.2
Build dir: /home/tkloczko/rpmbuild/BUILD/liburing-liburing-2.2/x86_64-redhat-linux-gnu
Build type: native build
Project name: liburing
Project version: 2.2
C compiler for the host machine: /usr/bin/gcc (gcc 12.2.1 "gcc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-1)")
C linker for the host machine: /usr/bin/gcc ld.bfd 2.39-3
C++ compiler for the host machine: /usr/bin/g++ (gcc 12.2.1 "g++ (GCC) 12.2.1 20220819 (Red Hat 12.2.1-1)")
C++ linker for the host machine: /usr/bin/g++ ld.bfd 2.39-3
Host machine cpu family: x86_64
Host machine cpu: x86_64
Run-time dependency threads found: YES
Compiler for C supports arguments -Wstringop-overflow=0: YES
Compiler for C supports arguments -Warray-bounds=0: YES
Checking for type "__kernel_rwf_t" : YES
Checking whether type "struct __kernel_timespec" has members "tv_sec", "tv_nsec" : YES
Checking whether type "struct open_how" has members "flags", "mode", "resolve" : YES
Checking if "statx" compiles: YES
Checking if "glibc_statx" compiles: YES
Checking for type "ucontext_t" : YES
Checking for function "makecontext" : YES
Configuring config-host.h using configuration
Configuring compat.h using configuration

man/meson.build:1:0: ERROR: File io_uring_prep_recv_multishot.3 does not exist.

[kloczek]

Patch fixing prev issue:

--- a/man/meson.build
+++ b/man/meson.build
@@ -19,6 +19,7 @@
   'io_uring_prep_accept.3',
   'io_uring_prep_accept_direct.3',
   'io_uring_prep_cancel.3',
+  'io_uring_prep_cancel64.3',
   'io_uring_prep_close.3',
   'io_uring_prep_close_direct.3',
   'io_uring_prep_connect.3',
@@ -48,9 +49,7 @@
   'io_uring_prep_readv2.3',
   'io_uring_prep_readv.3',
   'io_uring_prep_recv.3',
-  'io_uring_prep_recv_multishot.3',
   'io_uring_prep_recvmsg.3',
-  'io_uring_prep_recvmsg_multishot.3',
   'io_uring_prep_remove_buffers.3',
   'io_uring_prep_rename.3',
   'io_uring_prep_renameat.3',
@@ -77,13 +76,6 @@
   'io_uring_queue_exit.3',
   'io_uring_queue_init.3',
   'io_uring_queue_init_params.3',
-  'io_uring_recvmsg_cmsg_firsthdr.3',
-  'io_uring_recvmsg_cmsg_nexthdr.3',
-  'io_uring_recvmsg_name.3',
-  'io_uring_recvmsg_out.3',
-  'io_uring_recvmsg_payload.3',
-  'io_uring_recvmsg_payload_length.3',
-  'io_uring_recvmsg_validate.3',
   'io_uring_register.2',
   'io_uring_register_buffers.3',
   'io_uring_register_buf_ring.3',

More issues:

• after use tests=true meson fails with

test/meson.build:169:4: ERROR: File fd-pass.c does not exist.

• after use examples=true meson fails

examples/meson.build:16:4: ERROR: File io_uring-udp.c does not exist.

[eli-schwartz]

Cannot cleanly apply this PR on top of https://github.com/axboe/liburing//archive/liburing-2.2.tar.gz

On July 5 you said the same thing, and several people pointed out that this PR is based on the development version and isn't supposed to apply on top of a stable release.

For example, #622 (comment)

If you do not want to use the previously linked meson wrap for liburing-2.2 you can use my branch porting this meson PR to liburing-2.2.

To be quite honest, I'm highly surprised you're asking the same question again, in the same PR, for the same old version. Did you completely forget you asked it? Why do you even think that people submitting PRs or patch series' to a project would be submitting it for the frozen release tag, which cannot ever be accepted ever under any circumstances ever...

... rather than doing like literally every software developer, or external contributor, for any project anywhere, and targeting the development version?

Why do we need 3 different comments describing how something that isn't supposed to work, doesn't work? What's your goal here? If you're not here to waste everyone's time, why not just use the version you were linked to a couple months ago?

[kloczek]

To be quite honest, I'm highly surprised you're asking the same question again, in the same PR, for the same old version. Did you completely forget you asked it? Why do you even think that people submitting PRs or patch series' to a project would be submitting it for the frozen release tag, which cannot ever be accepted ever under any circumstances ever...

Because this PR still is in in opened state.