Lambda Function Unable To Find Header Files

[njpau]

Context:
I'm quite new to AWS and am go illiterate as well (for now). My current aim is to reproduce the experiment from this blog so that I can see how llama works and then hand it off to an intern to do more research to understand how we can integrate llama into our current meson-and-ninja based build system to build our firmware applications. However, in my efforts to build LLVM, I hit the following type of errors:
[98/2572] Building CXX object lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o /home/njpau/go/bin/llamac++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support -I/home/njpau/git/llvm-project/llvm/lib/Support -I/usr/include/libxml2 -Iinclude -I/home/njpau/git/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O0 -fno-exceptions -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o -c /home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp _root/home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp:16:28: fatal error: llvm/ADT/Twine.h: No such file or directory #include "llvm/ADT/Twine.h" ^ compilation terminated. Running llamacc: invoke: exit 1

Everything I have done today is from scratch, including setting up my free tier AWS account. These are the steps I've gone through:

1. Installed go on my 4C/8T Debian x86_64 GNU/Linux machine and set up an AWS account for myself.
2. Installed llama from source using the instructions in the Install llama section under Getting Started in the README. Did the llamac++ symlinking as well.
3. Installed awscli2 and my creds have been appropriately configured and stored in ~/.aws
4. Ran llama bootstrap. After completion, I went to my aws console and verified that the cloudformation stack had been successful created and that I had an empty S3 bucket, and empty ECR repo and an IAM role for basic lambda execution.
5. Ran home/njpau/go/src/github.com/nelhage/llama/scripts/build-gcc-image --local-headers to build my debian image and lambda function. Verified that I see 'gcc' image and function in my aws console under ECR and Lambda.
6. Cloned the llvm-project from github and checked-out the release/8.x branch (because I'm running python3.5 on my system)
7. cd llvm-project && mkdir -p build && cd build
8. Ran cmake -GNinja \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_ENABLE_PROJECTS=clang \ -DLLVM_TARGETS_TO_BUILD=X86 \ -DLLVM_PARALLEL_LINK_JOBS=8 \ -DLLVM_BUILD_TOOLS=OFF \ -DLLVM_BUILD_UTILS=OFF \ -DCMAKE_CXX_FLAGS_RELEASE="-O0" \ -DCLANG_ENABLE_STATIC_ANALYZER=OFF \ -DCLANG_ENABLE_ARCMT=OFF \ ../llvm/ to test building locally first.
9. Ran ninja build. Build successfully completed in roughly 20 mins.
10. Deleted the build dir, created a new one and cd-ed into it.
11. Ran LLAMACC_LOCAL=1 CC=llamacc CXX=llamac++ \ cmake -GNinja \ -DCMAKE_BUILD_TYPE=Release \ -DLLVM_ENABLE_PROJECTS=clang \ -DLLVM_TARGETS_TO_BUILD=X86 \ -DLLVM_PARALLEL_LINK_JOBS=8 \ -DLLVM_BUILD_TOOLS=OFF \ -DLLVM_BUILD_UTILS=OFF \ -DCMAKE_CXX_FLAGS_RELEASE="-O0" \ -DCLANG_ENABLE_STATIC_ANALYZER=OFF \ -DCLANG_ENABLE_ARCMT=OFF \ ../llvm/
12. Ran ninja build -j100. And that's when the build bombed out with 99 errors similar to the one above (i.e., reporting 'No such file or directory' for required headers).

What am I doing wrong here? Have I configured llama incorrectly? I have tried to follow the README + this blog post.

[nelhage]

Huh, that looks correct offhand. I can build fine on the release/8.3 branch on my desktop, so that's not the issue.

Can you try

LLAMACC_VERBOSE=1 ninja -v lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o

and paste the full output? That might give some better sense of what's going wrong.

[njpau]

njpau@debian-vm-1:~/git/llvm-project/build$ LLAMACC_VERBOSE=1 ninja -v lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o

[1/1] /home/njpaul/go/bin/llamac++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support -I/home/njpau/git/llvm-project/llvm/lib/Support -I/usr/include/libxml2 -Iinclude -I/home/njpau/git/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O0 -fno-exceptions -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o -c /home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp

FAILED: lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o /home/njpau/go/bin/llamac++ -DGTEST_HAS_RTTI=0 -D_GNU_SOURCE -D__STDC_CONSTANT_MACROS -D__STDC_FORMAT_MACROS -D__STDC_LIMIT_MACROS -Ilib/Support -I/home/njpau/git/llvm-project/llvm/lib/Support -I/usr/include/libxml2 -Iinclude -I/home/njpau/git/llvm-project/llvm/include -fPIC -fvisibility-inlines-hidden -Werror=date-time -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wwrite-strings -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment -fdiagnostics-color -ffunction-sections -fdata-sections -O0 -fno-exceptions -fno-rtti -MD -MT lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o -MF lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o.d -o lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o -c /home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cp

2021/07/13 13:13:35 run cpp -MM: ["c++" "-fPIC" "-fvisibility-inlines-hidden" "-Werror=date-time" "-std=c++11" "-Wall" "-Wextra" "-Wno-unused-parameter" "-Wwrite-strings" "-Wcast-qual" "-Wno-missing-field-initializers" "-pedantic" "-Wno-long-long" "-Wno-maybe-uninitialized" "-Wdelete-non-virtual-dtor" "-Wno-comment" "-fdiagnostics-color" "-ffunction-sections" "-fdata-sections" "-O0" "-fno-exceptions" "-fno-rtti" "-D" "GTEST_HAS_RTTI=0" "-D" "_GNU_SOURCE" "-D" "__STDC_CONSTANT_MACROS" "-D" "__STDC_FORMAT_MACROS" "-D" "__STDC_LIMIT_MACROS" "-I" "lib/Support" "-I" "/home/njpau/git/llvm-project/llvm/lib/Support" "-I" "/usr/include/libxml2" "-I" "include" "-I" "/home/njpau/git/llvm-project/llvm/include" "-M" "-MF" "-" "/home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp"]

2021/07/13 13:13:35 [llamacc] compiling remotely: daemon.InvokeWithFilesArgs{Trace:(*tracing.Propagation)(nil), Function:"gcc", ReturnLogs:false, Args:[]string{"c++", "-I", "_root/home/njpau/git/llvm-project/build", "-I", "_root/home/njpau/git/llvm-project/build/lib/Support", "-I", "_root/home/njpau/git/llvm-project/llvm/lib/Support", "-I", "_root/usr/include/libxml2", "-I", "_root/home/njpau/git/llvm-project/build/include", "-I", "_root/home/njpau/git/llvm-project/llvm/include", "-D", "GTEST_HAS_RTTI=0", "-D", "_GNU_SOURCE", "-D", "__STDC_CONSTANT_MACROS", "-D", "__STDC_FORMAT_MACROS", "-D", "__STDC_LIMIT_MACROS", "-c", "-o", "_root/home/njpau/git/llvm-project/build/lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o", "_root/home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp", "-MD", "-MF", "_root/home/njpau/git/llvm-project/build/lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o.d.tmp", "-fPIC", "-fvisibility-inlines-hidden", "-Werror=date-time", "-std=c++11", "-Wall", "-Wextra", "-Wno-unused-parameter", "-Wwrite-strings", "-Wcast-qual", "-Wno-missing-field-initializers", "-pedantic", "-Wno-long-long", "-Wno-maybe-uninitialized", "-Wdelete-non-virtual-dtor", "-Wno-comment", "-fdiagnostics-color", "-ffunction-sections", "-fdata-sections", "-O0", "-fno-exceptions", "-fno-rtti"}, Stdin:[]uint8(nil), Files:files.List{files.Mapped{Local:files.LocalFile{Path:"/home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp", Bytes:[]uint8(nil), Mode:0x0}, Remote:"_root/home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp"}}, Outputs:files.List{files.Mapped{Local:files.LocalFile{Path:"/home/njpau/git/llvm-project/build/lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o", Bytes:[]uint8(nil), Mode:0x0}, Remote:"_root/home/njpau/git/llvm-project/build/lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o"}, files.Mapped{Local:files.LocalFile{Path:"/home/njpau/git/llvm-project/build/lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o.d.tmp", Bytes:[]uint8(nil), Mode:0x0}, Remote:"_root/home/njpau/git/llvm-project/build/lib/Support/CMakeFiles/LLVMSupport.dir/AMDGPUMetadata.cpp.o.d.tmp"}}, DropSemaphore:true} _root/home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp:16:28: fatal error: llvm/ADT/Twine.h: No such file or directory

#include "llvm/ADT/Twine.h"
                           ^
compilation terminated.
Running llamacc: invoke: exit 1
ninja: build stopped: subcommand failed.

[nelhage]

Interesting, it's not picking up any of the headers. If you run

"c++" "-fPIC" "-fvisibility-inlines-hidden" "-Werror=date-time" "-std=c++11" "-Wall" "-Wextra" "-Wno-unused-parameter" "-Wwrite-strings" "-Wcast-qual" "-Wno-missing-field-initializers" "-pedantic" "-Wno-long-long" "-Wno-maybe-uninitialized" "-Wdelete-non-virtual-dtor" "-Wno-comment" "-fdiagnostics-color" "-ffunction-sections" "-fdata-sections" "-O0" "-fno-exceptions" "-fno-rtti" "-D" "GTEST_HAS_RTTI=0" "-D" "_GNU_SOURCE" "-D" "__STDC_CONSTANT_MACROS" "-D" "__STDC_FORMAT_MACROS" "-D" "__STDC_LIMIT_MACROS" "-I" "lib/Support" "-I" "/home/njpau/git/llvm-project/llvm/lib/Support" "-I" "/usr/include/libxml2" "-I" "include" "-I" "/home/njpau/git/llvm-project/llvm/include" "-M" "-MF" "-" "/home/njpau/git/llvm-project/llvm/lib/Support/AMDGPUMetadata.cpp"

by hand, what is the output?

[njpau]

Running that command by hand doesn't throw any compilation errors.

[njpau]

The only difference I can see, from what you've described on your blog post, is that I'm not passing the '-DLLVM_USE_LINKER=lld' argument to cmake, because I don't have lld. But then again, llvm still builds fine locally without that argument.

[nelhage]

That command should output a list of dependencies for AMDGPUMetadata.cpp. Is it doing so, or just returning with no error and no output? If there's no output, that suggests something might be wrong with your local compiler configuration.

[njpau]

Yep, it is just returning with no error and no output.
I haven't really changed anything WRT my local compiler configuration, I'm on a debian buster VM. Moreover, if there was something wrong with my local compiler config, wouldn't that have broken my local LLVM build too? I was able to build fine locally, without using llama.

[nelhage]

If I run that command in a Buster docker image, I see this:

root@d38a53f739fd:/src/build/fastbuild.llama# "g++" "-fPIC" "-fvisibility-inlines-hidden" "-Werror=date-time" "-std=c++11" "-Wall" "-Wextra" "-Wno-unused-parameter" "-Wwrite-strings" "-Wcast-qual" "-Wno-missing-field-initializers" "-pedantic" "-Wno-long-long" "-Wno-maybe-uninitialized" "-Wdelete-non-virtual-dtor" "-Wno-comment" "-fdiagnostics-color" "-ffunction-sections" "-fdata-sections" "-O0" "-fno-exceptions" "-fno-rtti" "-D" "GTEST_HAS_RTTI=0" "-D" "_GNU_SOURCE" "-D" "__STDC_CONSTANT_MACROS" "-D" "__STDC_FORMAT_MACROS" "-D" "__STDC_LIMIT_MACROS" "-I" "lib/Support" "-I" "/src/llvm/lib/Support" "-I" "/usr/include/libxml2" "-I" "include" "-I" "/src/llvm/include" "-M" "-MF" "-" "/src/llvm/lib/Support/AMDGPUMetadata.cpp"                                                                                                      
AMDGPUMetadata.o: /src/llvm/lib/Support/AMDGPUMetadata.cpp \                                                           
 /usr/include/stdc-predef.h /src/llvm/include/llvm/ADT/Twine.h \                                                       
 /src/llvm/include/llvm/ADT/SmallVector.h \          
 /src/llvm/include/llvm/ADT/iterator_range.h /usr/include/c++/8/iterator \                                             
 /usr/include/x86_64-linux-gnu/c++/8/bits/c++config.h \
 /usr/include/x86_64-linux-gnu/c++/8/bits/os_defines.h \
 /usr/include/features.h /usr/include/x86_64-linux-gnu/sys/cdefs.h \
 /usr/include/x86_64-linux-gnu/bits/wordsize.h \
 /usr/include/x86_64-linux-gnu/bits/long-double.h \
 /usr/include/x86_64-linux-gnu/gnu/stubs.h \
 /usr/include/x86_64-linux-gnu/gnu/stubs-64.h \
 /usr/include/x86_64-linux-gnu/c++/8/bits/cpu_defines.h \
 /usr/include/c++/8/bits/stl_iterator_base_types.h \
 /usr/include/c++/8/type_traits \
 /usr/include/c++/8/bits/stl_iterator_base_funcs.h \
 /usr/include/c++/8/bits/concept_check.h \
 /usr/include/c++/8/debug/assertions.h \
 /usr/include/c++/8/bits/stl_iterator.h \
 /usr/include/c++/8/bits/cpp_type_traits.h \
 /usr/include/c++/8/ext/type_traits.h /usr/include/c++/8/bits/move.h \
 /usr/include/c++/8/bits/ptr_traits.h /usr/include/c++/8/ostream \
 /usr/include/c++/8/ios /usr/include/c++/8/iosfwd \
 /usr/include/c++/8/bits/stringfwd.h /usr/include/c++/8/bits/memoryfwd.h \
 /usr/include/c++/8/bits/postypes.h /usr/include/c++/8/cwchar \
 /usr/include/wchar.h \
 /usr/include/x86_64-linux-gnu/bits/libc-header-start.h \
 /usr/include/x86_64-linux-gnu/bits/floatn.h \
 /usr/include/x86_64-linux-gnu/bits/floatn-common.h \
 /usr/lib/gcc/x86_64-linux-gnu/8/include/stddef.h \
 /usr/lib/gcc/x86_64-linux-gnu/8/include/stdarg.h \
 /usr/include/x86_64-linux-gnu/bits/wchar.h \
 /usr/include/x86_64-linux-gnu/bits/types/wint_t.h \
 /usr/include/x86_64-linux-gnu/bits/types/mbstate_t.h \
 /usr/include/x86_64-linux-gnu/bits/types/__mbstate_t.h \
 /usr/include/x86_64-linux-gnu/bits/types/__FILE.h \
 /usr/include/x86_64-linux-gnu/bits/types/FILE.h \
 /usr/include/x86_64-linux-gnu/bits/types/locale_t.h \
 /usr/include/x86_64-linux-gnu/bits/types/__locale_t.h \
 /usr/include/c++/8/exception /usr/include/c++/8/bits/exception.h \
 /usr/include/c++/8/bits/exception_ptr.h \
 /usr/include/c++/8/bits/exception_defines.h \
 /usr/include/c++/8/bits/cxxabi_init_exception.h \
[more headers]

Which is what -M -MF - is supposed to do.

What if you run echo '#include <iostream>' | g++ -M -MF - -xc++ /dev/stdin -- test out your compiler's -MF with a minimal test case. Does that generate any output?

[njpau]

Hmm, that's strange, running that command doesn't generate any output either.

[nelhage]

How did you get your buster VM? Is it a stock image somewhere I could easily reproduce so I can investigate? This is surprising behavior to me and smells obviously buggy, although – as you point out – the fact it can build llvm locally suggests it's mostly fine.

[njpau]

The VM is built using a preseeded iso that we have here in the office, and unfortunately I can't point you to a reproduceable image. Is there anything in particular that I can check out in my VM and send you outputs for?
Worst case scenario - i'll probably just try and make this work on a vm run from a standard debian stretch image.

[nelhage]

What is your C compiler? which g++ / g++ --version?

I've never seen this issue before and am a bit stumped offhand about what might be the issue.

[njpau]

g++ (Debian 6.3.0-18+deb9u1) 6.3.0 20170516
Copyright (C) 2016 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[nelhage]

Hmm, that's not the normal buster gcc, which should be 8.x: https://packages.debian.org/buster/gcc

That looks to be an oldstable version: https://packages.debian.org/stretch/gcc-6

I'm still surprised it doesn't appear to support -M properly, but are you able to upgrade gcc and try again?

[nelhage]

Oh gosh, I just grabbed a docker image with that version of gcc, and it seems that -MF - on that version writes to the literal file -, instead of stdout. I can easily work around that.

[njpau]

Thanks for the fix @nelhage ! I am now able to build using llama without any difficulties :)