Fix cross build #1174

nicowilliams · 2023-06-22T20:54:55Z

Let's see if this builds on Windows, and then let's see if we can do a cross-build w/ MingW at least part of the way through. To do a cross-build one has to first build native, then make install, then manually install slc, compile_et, and asn1_compile (yes, I mean to fix that so they get installed) into a bindir that will be referenced in ./configure --host=... --with-cross-tools=that-bindir ....

What this PR does:

fix use/non-use of ASN1_COMPILE_DEP makefile macro
import and use AX_PROG_CC_FOR_BUILD macro (it's broken in some versions of autoconf-archive)
add --disable-readline and --disable-editline for cross-compilation for Windows using MingW
check for more C types in configure
uses $(CC_FOR_BUILD) to build include/bits and use it to build krb5-types.h when cross-compiling
stop using AC_TYPE_PID_T and AC_TYPE_UID_T -- just check for those types and let include/bits deal with their absense
uses AX_COMPILE_CHECK_SIZEOF instead of AC_CHECK_SIZEOF (the former does not run the code it generates and compiles)
- installs autoconf-archive so AX_COMPILE_CHECK_SIZEOF can be used
causes heim_compile_et to get installed

This cross-compilation instructions then would be:

./configure --prefix=/tmp/h5l ... (or similar install location)
mkdir /tmp/cross-tools
cp -p /tmp/h5l/bin/asn1_compile /tmp/h5l/bin/heim_compile_et /tmp/h5l/libexec/slc /tmp/cross-tools/
cp -p /tmp/h5l/bin/heim_compile_et /tmp/cross-tools/compile_et
in a different build directory: ./configure --with-cross-tools=/tmp/cross-tools --host=... ...

nicowilliams · 2023-06-22T20:55:08Z

@riastradh

nicowilliams · 2023-06-23T02:33:14Z

Argh, no, include/krb5-types.h can't include config.h, so it has to be generated, but if we generate it then the program that generates it needs to be run in the build, so it has to be built with $(CC_FOR_BUILD), but that can't include config.h (or I can't get it to anyways).

riastradh · 2023-06-23T11:45:08Z

Argh, no, include/krb5-types.h can't include config.h, so it has to be generated, but if we generate it then the program that generates it needs to be run in the build, so it has to be built with $(CC_FOR_BUILD), but that can't include config.h (or I can't get it to anyways).

If you need particular configure-detected variable substitutions in include/krb5-types.h, which is all that config.h has, why can't you do it with AC_SUBST?

riastradh · 2023-06-23T13:29:47Z

roken.h appears to have the same issue, and it is a public header file. It is generated by roken-h-process.pl but that script appears to be somewhat broken in a way I haven't diagnosed yet:

perl ../../../../cross/cf/roken-h-process.pl \
-c ../../include/config.h  \
-p ../../../../cross/lib/roken/roken.h.in -o roken.h
gmake  all-am
gmake[3]: Entering directory '/home/riastradh/crypto/krb/heimdal/build/aarch64--netbsd10/lib/roken'
  CC       rkvis-vis.o
In file included from ../../../../cross/lib/roken/vis.c:60:0:
./roken.h:147:13: error: conflicting types for 'ssize_t'
 typedef int ssize_t;
             ^~~~~~~

For some reason even though HAVE_SSIZE_T is defined in config.h, roken-h-process.pl doesn't figure it out. Passing a file with the output of aarch64--netbsd-gcc -dM -x c -E config.h instead of passing config.h itself appears to fix it.

That said, I suspect it will be better to define autoconf substitutions to generate both krb5-types.h and roken.h uniformly with a simple templating mechanism, and ditch the Perl script that tries to heuristically compute a partial evaluation of the cpp directives in roken.h.in.

riastradh · 2023-06-23T14:09:32Z

lib/asn1/Makefile.am

+if !CROSS_COMPILE
+$(ASN1_COMPILE_DEP): asn1_compile
+endif
+


Confused, what does this do? Was this supposed to be a variable assignment, ASN1_COMPILE_DEP = asn1_compile?

(Might want asn1_compile$(BUILD_EXEEXT) too, with BUILD_EXEEXT = @BUILD_EXEEXT@ substituted by configure through AX_PROG_CC_FOR_BUILD in case building on Windows is ever relevant here. Not important to me but it might bite you later if it is important to you.)

Ah, this is because ASN1_COMPILE_DEP expands to a relative path with ..s in it and make doesn't seem to understand that it's a path to ./asn1_compile. This was a hack I did to make it work. Because we use a recursive makefile setup, we could just only set ASN1_COMPILE_DEP = asn1_compile and use that only in lib/asn1/Makefile.am, and in lib/hdb/Makefile.am not use it at all.

IOW, this hack convinces make that it's built the thing (asn1_compile) that targets depend on via a relative path.

nicowilliams · 2023-06-23T16:10:27Z

Argh, no, include/krb5-types.h can't include config.h, so it has to be generated, but if we generate it then the program that generates it needs to be run in the build, so it has to be built with $(CC_FOR_BUILD), but that can't include config.h (or I can't get it to anyways).

If you need particular configure-detected variable substitutions in include/krb5-types.h, which is all that config.h has, why can't you do it with AC_SUBST?

I'd need to have every macro that uses AC_DEFINE to also make it an AC_SUBST, right? How would I do that, or is it automatic anyways? Could I overwrite AC_DEFINE to also do AC_SUBST?

nicowilliams · 2023-06-23T16:16:39Z

Trying it out, no AC_DEFINE does not imply AC_SUBST.

nicowilliams · 2023-06-23T16:56:41Z

Ah, I think I got it: build include/bits with $(CC_FOR_BUILD) and add -I$(top_builddir)/include so it can include config.h.

nicowilliams · 2023-06-23T21:27:32Z

This is a bit inefficient -- it'd be better to not build compile_et and heim_compile_et, but to only build heim_compile_et and then change the autoconf code so that the relevant macro get set to heim_compile_et when using the in-tree com_err.

We should also have a GitHub Action workflow that tests cross-compilation.

riastradh · 2023-06-23T22:05:31Z

gmake[2]: Entering directory '/home/riastradh/crypto/krb/heimdal/build/aarch64--netbsd10/lib/roken'
perl ../../../../cross/cf/roken-h-process.pl \
-c ../../include/config.h  \
-p ../../../../cross/lib/roken/roken.h.in -o roken.h
gmake  all-am
gmake[3]: Entering directory '/home/riastradh/crypto/krb/heimdal/build/aarch64--netbsd10/lib/roken'
  CC       rkvis-vis.o
In file included from ../../../../cross/lib/roken/vis.c:60:0:
./roken.h:147:13: error: conflicting types for 'ssize_t'
 typedef int ssize_t;
             ^~~~~~~
In file included from ../../include/config.h:1652:0,
                 from ../../../../cross/lib/roken/vis.c:59:
/usr/include/sys/types.h:284:24: note: previous declaration of 'ssize_t' was here
 typedef _BSD_SSIZE_T_  ssize_t;
                        ^~~~~~~

Examination of roken.h, and of the output of roken-h-process.pl, shows that it has treated essentially all the macros as undefined, despite things like #define HAVE_SSIZE_T 1 in config.h.

nicowilliams · 2023-06-23T22:24:37Z

That's on NetBSD? Yeah, I've not tackled lib/roken yet.

nicowilliams · 2023-06-23T22:25:09Z

Do you know of a way to run NetBSD builds in GitHub Actions?

riastradh · 2023-06-23T22:52:42Z

Do you know of a way to run NetBSD builds in GitHub Actions?

I don't think GHA supports NetBSD, but you can easily do a NetBSD cross-build on your own system as long as it's reasonably POSIXish (*BSD, GNU/Linux, macOS, possibly even Cygwin or similar):

git clone https://github.com/NetBSD/src
cd src
./build.sh -O ../obj -T ../tools -U -u -m evbarm64 -j4 distribution

Then obj/destdir.evbarm is a sysroot, and tools/bin/ has the cross-compiler toolchain (aarch64--netbsd-gcc and stuff).

riastradh · 2023-06-23T22:55:01Z

Apparently there is this, according to random web searching: https://github.com/marketplace/actions/netbsd-vm (No idea who set it up or whether it works.)

nicowilliams · 2023-06-24T03:58:44Z

Since the only things I've ever cross-built are for MingW, I'm trying that. One of the first things I ran into is the need to modify configure.ac so I can disable use of libedit.

nicowilliams · 2023-06-24T04:50:01Z

Well, this is fun:

PATH="/home/build/mingw/mingw-w64-x86_64/bin:$PATH" make V=1 bits
gcc -o bits -DHAVE_CONFIG_H -I../include /home/nico/ws/heimdal/include/bits.c
In file included from /home/nico/ws/heimdal/include/bits.c:37:
../include/config.h:1589:15: error: two or more data types in declaration specifiers
 1589 | #define gid_t int
      |               ^~~
../include/config.h:1616:15: error: two or more data types in declaration specifiers
 1616 | #define uid_t int
      |               ^~~
make: *** [Makefile:1324: bits] Error 1

I need to include config.h in bits.c so bits can know about the host, but I can't include config.h because some of its contents conflicts with the build OS's includes.

There has to be a better way to do this...

EDIT: At least I can see that gid_t and uid_t are the only problems there at this time, and if I fix those then include/bits builds and runs and seems to work correctly. Though this whole approach of using a C program built for the build system that includes the host system's config.h is just bound to be brittle.

EDIT: Still, I might be able to commit an include/config.h.in that excludes those problematic lines and have bits emit proper typedefs for gid_t and uid_t, and that then might work for now. Or drop AC_REQUIRE([AC_TYPE_UID_T]), right, as that's where these defines must be coming from.

EDIT: Not using AC_TYPE_UID_T gets me past getting include/krb5-types.h built on Linux for MingW. Progress. [A MingW build would be fantastic.]

Some versions of autoconf-archive have a broken AX_PROG_CC_FOR_BUILD.

In order to use `./configure --host=... --with-tools=DIR ...` one needs a directory in which to find: `slc`, `compile_et`, and `asn1_compile`. `slc` already gets installed into `${libexecdir}/`, and `asn1_compile` gets installed into `${bindir}/`, but `compile_et` does not and should not get installed because it would conflict with the system's com_err. So let's install a `heim_compile_et` into `${bindir}`.

nicowilliams · 2023-06-24T16:54:18Z

Success! Built on my x86 laptop:

🥳

Started putting a wip/heimdal-git into pkgsrc, and amusingly, it appears that -- due to doxygen -- Heimdal requires not only its own ASN.1 compiler/library, but another ASN.1 compiler/library, libtasn1, via p11-kit via gnutls via libcups via ghostscript, which doxygen depends on.

Oh my. Well, and OpenSSL has its own ASN.1 stack (but no compiler, just error-prone hand-coded translations of ASN.1 modules to C using a bunch of macros).

IMO Heimdal's is the nicest ASN.1->C compiler out there.

nicowilliams · 2023-06-24T16:56:12Z

A mingw build will need some code in lib/Makefile.am to make a consolidated library like the native Windows build does.

BTW @riastradh thanks for the clues on how to make cross-compilation work again!

riastradh · 2023-06-24T19:56:58Z

Some quirks currently prevent an installed Heimdal from working with --with-cross-tools:

slc is installed in libexec/heimdal, but the other tools are installed in bin. This change was made in 5c93af5, but the commit message doesn't explain why. I wonder whether it was intentional?
asn1_compile is installed as such, but compile_et is now installed as heim_compile_et instead of compile_et. However, configure expects both to be called asn1_compile and compile_et, with no heim_ prefix.
The logic to set ASN1_COMPILE, SLC, and ac_cv_prog_COMPILE_ET is inconsistent (why ac_cv_prog_ just for compile_et?) and provides no way to override it on the command line or in the environment. Having the script set ac_cv_prog_COMPILE_ET interferes with the cache semantics of AC_CHECK_PROG in CHECK_COMPILE_ET. Unclear whether CHECK_COMPILER_ET is really that important these days (and if it is, why it's not important to do the same for asn1_compile and slc).

Side note: make install installs a lot of stuff in various places, for very different purposes -- if I'm just installing Heimdal for bin/kinit and bin/klist and other client-side SSO tools, I also get bin/bsearch, bin/idn-lookup, bin/otp, lib/ipc_csr_authorizor.la, man/man3/ecalloc.3, and various other things that aren't obviously related to client-side SSO. Is there any guidance on how this stuff could/should be split up? Of course if there's interdependencies (like bin/kinit presumably isn't useful without lib/libroken.la) it can't be split up, but is lib/test_negoex_mech.la useful for that? (And does something that's not libfoo.la even belong under lib?)

nicowilliams · 2023-06-25T01:11:06Z

Some quirks currently prevent an installed Heimdal from working with --with-cross-tools:

1. slc is installed in libexec/heimdal, but the other tools are installed in bin.  This change was made in [5c93af5](https://github.com/heimdal/heimdal/commit/5c93af553b7aad47af641401dc982e755123a159), but the commit message doesn't explain why.  I wonder whether it was intentional?

I would like others to discover and use asn1_compile, so ${bindir} it is for it. slc was getting installed into libexec/heimdal/ from before. I suppose we could move it to ${bindir}. I'm not sure if there would immediately be conflicts with other things in ${bindir}. I'm also not sure if anyone ought to discover slc/libsl -- it's a command-line parsing utility for programs with sub-commands -- there's many of these things out there, but not very many ASN.1 compilers.

2. asn1_compile is installed as such, but compile_et is now installed as heim_compile_et instead of compile_et.  However, configure expects both to be called asn1_compile and compile_et, with no heim_ prefix.

It can't be called compile_et because that is a conflict. I meant to, but didn't get to, change it so Heimdal invokes the in-tree et compiler as heim_compile_et.

3. The logic to set ASN1_COMPILE, SLC, and ac_cv_prog_COMPILE_ET is inconsistent (why ac_cv_prog_ just for compile_et?) and provides no way to override it on the command line or in the environment.  Having the script set ac_cv_prog_COMPILE_ET interferes with the cache semantics of AC_CHECK_PROG in CHECK_COMPILE_ET.  Unclear whether CHECK_COMPILER_ET is really that important these days (and if it is, why it's not important to do the same for asn1_compile and slc).

compile_et will be already installed on the host system if com_err is installed. Heimdal has its own private version of com_err for when com_err is not available on the host. That's why we AC_CHECK_PROG for compile_et.

As for the other two, slc and asn1_compile, those are unique to Heimdal -- there are no work-alike alternatives elsewhere. The various other ASN.1->C compilers that exist out there are not remotely compatible with Heimdal's in their emitted C. Nor are we likely to dump slc/libsl and replace it with something else given how much work that would be. Therefore these can't be checked for. The host could have Heimdal installed already, but we don't commit to either of these two compilers (slc and asn1_compile) being backwards-compatible, so we would prefer to use the in-tree versions to any older version of Heimdal that might be installed.

There is a separate problem with libtool where trying to build Heimdal with internal ABI changes fails if there's a Heimdal already installed at the desired ${prefix}. I don't know how to make libtool never look in ${prefix} for Heimdal libraries to link with -- basically, we need libtool to put -L${top_builddir}/... ahead of any -L${libdir}, and we need libtool --mode=execute ... to pick up the objects/archives from ${top_buildir}/... before any from ${libdir}. Any ideas?

Side note: make install installs a lot of stuff in various places, for very different purposes -- if I'm just installing Heimdal for bin/kinit and bin/klist and other client-side SSO tools, I also get bin/bsearch, bin/idn-lookup, bin/otp, lib/ipc_csr_authorizor.la, man/man3/ecalloc.3, and various other things that aren't obviously related to client-side SSO. Is there any guidance on how this stuff could/should be split up? Of course if there's interdependencies (like bin/kinit presumably isn't useful without lib/libroken.la) it can't be split up, but is lib/test_negoex_mech.la useful for that? (And does something that's not libfoo.la even belong under lib?)

If you build with dynamic linking you can use ldd to work out the dependencies for the things you want to install. If you build statically then you can just use the executables you want. I don't have BSD packaging to share, but basically if I were going to write a set of packages for Heimdal I'd have:

heimdal-base, with libbase and libroken and libwind
heimdal-base-dev, with compilation links and headers
heimdal-asn1, with libasn1 and asn1_print
heimdal-asn1-dev, with asn1_compile and compilation link for libasn1 and headers
heimdal-hx509, with libhx509 and hxtool
heimdal-hx509-dev, with libhx509's compilation link and headers
heimdal-krb5 with libkrb5, kinit, heimtool, ..., and maybe also include all the GSS-API in this pkg, or maybe also have a heimdal-gss-api pkg suite?
heimdal-krb5-dev, ...
heimdal-kdc, with the KDC daemons and utilities and plugins like ipc_csr_authorizer.so
heimdal-kadm5, with libkadm5clnt and libkadm5srv
heimdal-kadm5-dev, ...
heimdal-kadmin, with kadmin and kadmind, and maybe httpkadmind

Add -doc variants for docs.

As you might have noticed, Heimdal is not just about Kerberos. It has: an ASN.1 compiler and library, an x.509 library and command-line tool, a krb5 API fairly close to MIT's, a GSS-API implementation, a libkadm5clnt and libkadm5srv that are MIT-like, various Kerberos-related daemons including a KDC, and online CA, the otp stuff, and more. Heimdal has an x.509 stack because Kerberos w/ PKINIT requires one. The ASN.1 stack is, of course, useful for both, Kerberos and x.509.

The otp stuff I think we should really remove from the tree. Or rework it. The otp stuff is basically unusable at this time.

If there is a way to describe this generically for a bunch of packaging systems, I'd love to see it and include an instance in-tree.

Add lib/roken/getline.c based on the in-tree libedit, with slight alterations: - grow buffer by 1.5x (+ 16 bytes) when needed, not 4KB - don't use SSIZE_MAX as the max line size; use 32KB instead

nicowilliams · 2023-06-25T03:35:44Z

There is at least one suspect commit here, c5933ad cf: Lose some AC_FIND_IF_NOT_BROKEN()s (suspect) (EDIT: Yup, that's a no-good commit). I need to figure out if there's a better way to handle these, but I had to comment these out to get things to build with Mingw.

Currently I'm able to build up to lib/roken with Mingw. One annoying thing is that the Mingw gcc yields errors when one uses printf format specifiers with ll (e.g., %llu), and so we have to switch to using PRIu64 and PRId64, and that's probably going to be messy as there's 131 of those in-tree right now, and quite a few of those are in SQLite3. That probably just means I need to upgrade my Mingw installation that I've been using for... years.

We don't use RW locks in Heimdal, and we should never use RW locks in Heimdal because RW locks are terrible. Instead we should import a user-land RCU library if ever we need RW locks. TODO: Just remove all RW locks functions/macros in include/heim_threads.h.

nicowilliams · 2023-06-25T04:57:46Z

I'm getting past lib/roken with my unpushed changes, and, indeed, I'm seeing that the configure logic for dealing with compile_et is busted.

nicowilliams · 2023-06-25T05:22:44Z

Ah right, the PRIu64 thing, if we're using the UCRT then we're OK, or if we're using Heimdal's snprintf() we're ok, so I need to arrange for either when cross-building with Mingw.

riastradh · 2023-06-26T13:26:38Z

Current state of the branch at 808ecbe fails in a native build for me with this error:

gmake[3]: Entering directory '/home/riastradh/pkgsrc/current/cross/work/wip/heimdal-git/work/heimdal/lib/roken'
...
ld: ./.libs/libroken.so: undefined reference to `rk_wait_for_process'
gmake[3]: *** [Makefile:1399: rkbase32] Error 1

Looks like you defined simple_exec_c but referenced simple_exec.

nicowilliams · 2023-06-26T15:27:12Z

Looks like you defined `simple_exec_c` but referenced `simple_exec`.

Yup, that's fixed in my unpushed changes. I also need to fix the Windows build. That means I need to separate all my mingw changes. I probably won't get to it today.

asn1: Use ASN1_COMPILE_DEP macro

ddadcef

nicowilliams force-pushed the nico/fix_cross_build branch 2 times, most recently from dffdae4 to d232fa6 Compare June 23, 2023 02:27

riastradh reviewed Jun 23, 2023

View reviewed changes

nicowilliams force-pushed the nico/fix_cross_build branch from d232fa6 to 68fc6f1 Compare June 23, 2023 16:56

nicowilliams marked this pull request as ready for review June 23, 2023 21:11

nicowilliams added 9 commits June 24, 2023 00:17

cf: Import AX_PROG_CC_FOR_BUILD macro

4d0d834

Some versions of autoconf-archive have a broken AX_PROG_CC_FOR_BUILD.

cf: Let include/bits deal with missing gid_t/uid_t

af0b95c

cf: Add --disable-{read,edit}line options

fccc437

cf: Use AX_PROG_CC_FOR_BUILD

f676979

cf: Don't use AC_CHECK_SIZEOF

2081a4b

cf: Also check sizes of short, int, long, long long

d1004a2

include: Fix cross-compilation of bits, krb5-types.h

47da6a4

GHA: Add autoconf-archive dependency for AX macros

6cab3aa

nicowilliams added 10 commits June 24, 2023 22:20

cf: Check for minwindef.h (Mingw)

aaf76fd

roken: Prepare for Mingw

28df1d8

include: Make bits cross-build with Mingw

605dcc6

w32: Define HAVE_WINSOCK2_H and HAVE_WS2TCPIP_H

81a4286

roken: Add getdelim()/getline() (from libedit)

051a740

Add lib/roken/getline.c based on the in-tree libedit, with slight alterations: - grow buffer by 1.5x (+ 16 bytes) when needed, not 4KB - don't use SSIZE_MAX as the max line size; use 32KB instead

cf: Add MINGW support (partial)

1e1cf37

cf: We no longer need daemon()

f11f278

roken: Move getifaddrs_w32.c into getifaddrs.c

1ef028c

roken: Add MINGW support (partial)

768b958

roken: Add MINGW support (rest)

de7e754

nicowilliams force-pushed the nico/fix_cross_build branch from 483681f to c5933ad Compare June 25, 2023 03:28

nicowilliams added 4 commits June 24, 2023 23:29

fixup! roken: Add MINGW support (rest)

fbebfab

roken: Windows lacks EAI_SYSTEM

18d48e1

roken: On Windows gai_strerror() returns char *

3c94246

nicowilliams added 4 commits June 25, 2023 00:11

cf: Fix compile_et cross-build logic

9e17f3b

fixup! roken: On Windows gai_strerror() returns char *

7b4d179

roken: Add missing get(e|)[ug]id() function prototypes

b98ec04

base: Mingw

99dd68b

cf,roken: Mingw needs an fsync()

808ecbe

nicowilliams force-pushed the nico/fix_cross_build branch from c5933ad to 808ecbe Compare June 25, 2023 05:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix cross build #1174

Fix cross build #1174

nicowilliams commented Jun 22, 2023 •

edited

Loading

nicowilliams commented Jun 22, 2023

nicowilliams commented Jun 23, 2023

riastradh commented Jun 23, 2023

riastradh commented Jun 23, 2023 •

edited

Loading

riastradh Jun 23, 2023

nicowilliams Jun 23, 2023 •

edited

Loading

nicowilliams commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

riastradh commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

riastradh commented Jun 23, 2023

riastradh commented Jun 23, 2023

nicowilliams commented Jun 24, 2023

nicowilliams commented Jun 24, 2023 •

edited

Loading

nicowilliams commented Jun 24, 2023

nicowilliams commented Jun 24, 2023 •

edited

Loading

riastradh commented Jun 24, 2023

nicowilliams commented Jun 25, 2023 •

edited

Loading

nicowilliams commented Jun 25, 2023 •

edited

Loading

nicowilliams commented Jun 25, 2023

nicowilliams commented Jun 25, 2023

riastradh commented Jun 26, 2023

nicowilliams commented Jun 26, 2023 via email

Fix cross build #1174

Are you sure you want to change the base?

Fix cross build #1174

Conversation

nicowilliams commented Jun 22, 2023 • edited Loading

nicowilliams commented Jun 22, 2023

nicowilliams commented Jun 23, 2023

riastradh commented Jun 23, 2023

riastradh commented Jun 23, 2023 • edited Loading

riastradh Jun 23, 2023

Choose a reason for hiding this comment

nicowilliams Jun 23, 2023 • edited Loading

Choose a reason for hiding this comment

nicowilliams commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

riastradh commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

nicowilliams commented Jun 23, 2023

riastradh commented Jun 23, 2023

riastradh commented Jun 23, 2023

nicowilliams commented Jun 24, 2023

nicowilliams commented Jun 24, 2023 • edited Loading

nicowilliams commented Jun 24, 2023

nicowilliams commented Jun 24, 2023 • edited Loading

riastradh commented Jun 24, 2023

nicowilliams commented Jun 25, 2023 • edited Loading

nicowilliams commented Jun 25, 2023 • edited Loading

nicowilliams commented Jun 25, 2023

nicowilliams commented Jun 25, 2023

riastradh commented Jun 26, 2023

nicowilliams commented Jun 26, 2023 via email

nicowilliams commented Jun 22, 2023 •

edited

Loading

riastradh commented Jun 23, 2023 •

edited

Loading

nicowilliams Jun 23, 2023 •

edited

Loading

nicowilliams commented Jun 24, 2023 •

edited

Loading

nicowilliams commented Jun 24, 2023 •

edited

Loading

nicowilliams commented Jun 25, 2023 •

edited

Loading

nicowilliams commented Jun 25, 2023 •

edited

Loading