Create a compile_commands database for a cxx binary and its dependencies

[lvaroqui]

Hello!

I set up a little C++ project in order to test Buck2 (I don't have any experience with Buck or Bazel).

One of the first thing I tried to do is setting up the generation of a compile_commands database for my IDE. I found the sub_target compile_commands but it seems it only allows generating the db for the binary and not its dependencies. Is it possible to do so? Also, I have to ask explicitly for the sub_target, but I would prefer to always update the db when the binary is updated, is this possible?

Thanks for your help!

[ndmitchell]

There is a compilation-database subtarget on cxx_library and cxx_binary. Internally at Meta we have an LSP mode that when a file is opened, it uses the owners query (through Bxl) to find the right target, grabs the [compilation-database] for that target, and then uses that to drive the LSP. We haven't plumbed any of that into open source pieces. How do existing C++ LSP servers get the compilation database? If that is configurable, then it probably isn't too hard to hook it up to what Buck2 provides. But I don't think anyone has tried yet.

[yeswalrus]

Chiming in since I'd love to see this (it's a huge pain in bazel). Generally speaking, clangd looks it up in a single monolithic compile_commands.json. Handling headers can be a challenge in that context.
For good clangd support, you'd want to aggregate all the compilation database files together output to the source root, and include entries for both source & header files. See https://github.com/hedronvision/bazel-compile-commands-extractor/blob/main/ImplementationReadme.md for a detailed explanation of what's required (and some technical details about how it happens in bazel)

The approach you describe, if I understand correctly, would fail for header-only libraries - Headers need entries in the database with compile commands for compiling any source file which actually includes them, which can be a bit challenging to determine, and which just using the compile commands for the target doesn't quite accomplish, at least not in the general case

[cpsauer]

^ Author of that Bazel tool here. Can confirm it was a pain :)

(Arriving not because backlinks are my batsignal or something; I just happened to be checking out the status of buck2.)

@ndmitchell, thanks for all you guys are doing here to share this with the world. Could I add to the header-only question: How do you guys handle the case where a parent target configures its children? For example, where an iOS rule configures cxx_libraries to cross compile?

[photex]

I use clangd a lot as my C++ lsp, and previously only ever configured it for a single compilation database. But after looking into their issue tracker a bit, I saw that it was pointed out that you can alternatively configure clangd by paths in a repo:
https://clangd.llvm.org/config.html#if

Clangd also does a bit of guesswork if it doesn't find a file in a configured compilation database: https://clangd.llvm.org/faq#what-can-i-do-if-clangd-chooses-the-wrong-source-file-to-infer-commands-for-a-header

with Bazel I was typically generating one using the hedronvision tool, and then post processing it a bit.

[cpsauer]

@photex, about that post processing: Anything we should learn from, there? We're always keen to just have the tool automatically clean up the command for folks.

[photex]

If I really think about it, I think the post processing stopped being necessary in the most recent versions. But basically if the database wasn’t comprehensive enough I’d use the compdb script to fill it out.

https://pypi.org/project/compdb/

Unrelated to Bazel, but some Xcode projects at work also required me to fix toolchain paths and header paths. That’s not really an issue for buck2 though! :)

[cpsauer]

Thanks for replying so quickly! Compdb just to add commands headers--or for something else? (The tool should already be auto-enriching with headers, and way faster bc it can hit cache, but I definitely want to know if there's something we're missing.)

Curious also if there are more generalizable things we should fix for those Xcode projects. If so, would love more details (either here, or in an issue over on our project to not overly hijack this thread).

[andyfriesen]

I've been fiddling with solutions to this problem. A simple thing you can do is to just write a little script and build rule that combines databases: https://gist.github.com/andyfriesen/731baabd7cec5d1c24c53cc0691ca249

You can see that it has some problems: You need to name each individual target that you'd like to combine. It's also pretty slow. I'm testing this on a project with about 400 source files and it takes about 2s to build the database up from nothing.

BXL seems like it ought to be the solution here as it's super trivial to query the system for all cxx_binary and cxx_library rules, but I'm finding it quite tricky to get at the actual commandline arguments that are passed to the compiler from that angle.

If that's possible, it should be possible to write a single BXL that generates the whole database in a single action. This should be quite a lot faster and mostly automatic.