How do I depend on Buck2 targets from another project?

[cbarrete]

Hello, I'm having trouble depending on the buck2 repo via external cells.

I have the following setup, informed by the documentation:

.buckconfig

[repositories]
root = .
prelude = prelude
toolchains = toolchains
buck2 = buck2
none = none

[repository_aliases]
config = prelude
ovr_config = prelude
fbcode = none
fbsource = none
fbcode_macros = none
buck = none

[parser]
target_platform_detector_spec = target:root//...->prelude//platforms:default

[external_cells]
prelude = bundled
buck2 = git

[external_cell_buck2]
git_origin = https://github.com/facebook/buck2
commit_hash = 681c0d24a76376c75645f6a9dd633790f5f5910e

(The URL and git commit hash are correct)

BUCK

rust_binary(
    name = "main",
    srcs = ["main.rs"], # Just a hello world, nothing fancy.
    deps = [
        "buck2//app/buck2_test_runner:buck2_test_runner",
    ],
)

I'm getting errors when running various commands:

> buck2 targets buck2//...
File changed: root//4913
File changed: root//main.rs
File changed: root//main.rs~
Error resolving recursive spec `buck2///...`

Caused by:
    0: Error listing dir ``
    1: Error listing directory
    2: read_dir(/path/to/buck2-external-cell-test/buck-out/v2/external_cells/git/681c0d24a76376c75645f6a9dd633790f5f5910e)
    3: No such file or directory (os error 2)
Build ID: 896501c5-1503-4b38-bd19-22eb16de01c3
Jobs completed: 2. Time elapsed: 0.0s.

> buck2 expand-external-cell buck2
File changed: root//4913
File changed: root//BUCK
File changed: root//BUCK~
Error copying from src path `/path/to/buck2-external-cell-test/buck-out/v2/external_cells/git/681c0d24a76376c75645f6a9dd633790f5f5910e` to dest path `/path/to/buck2-external-cell-test/buck2`

Caused by:
    0: symlink_metadata(/path/to/buck2-external-cell-test/buck-out/v2/external_cells/git/681c0d24a76376c75645f6a9dd633790f5f5910e)
    1: No such file or directory (os error 2)
Build ID: 2e48da4b-3020-42b0-96dc-8a73c150f2ad
Jobs completed: 2. Time elapsed: 0.0s.

> buck2 build :
Build ID: 998f4c90-d5ad-47fa-9fb5-c8b72d252792
Jobs completed: 51. Time elapsed: 0.1s.
BUILD FAILED
Error running analysis for `root//:main (prelude//platforms:default#904931f735703749)`

Caused by:
    0: Error in configured node dependency, dependency chain follows (-> indicates depends on, ^ indicates same configuration as previous):
              root//:main (prelude//platforms:default#904931f735703749)
           -> buck2//app/buck2_test_runner:buck2_test_runner (^)
       
    1: looking up unconfigured target node `buck2//app/buck2_test_runner:buck2_test_runner`
    2: Error loading targets in package `buck2//app/buck2_test_runner` for target `buck2//app/buck2_test_runner:buck2_test_runner`
    3: gathering package listing for `buck2//app/buck2_test_runner`
    4: Error listing dir ``
    5: Error listing directory
    6: read_dir(/path/to/buck2-external-cell-test/buck-out/v2/external_cells/git/681c0d24a76376c75645f6a9dd633790f5f5910e)
    7: No such file or directory (os error 2)

When I inspect buck-out/v2/external_cells/git, it is empty, so I'm not sure what to try next.

Ideally, could we get a full example that shows how to use external cells?

Also, is building buck2 using buck2 using the open source build and codebase supported? The documentation only mention building it with cargo and doing a quick buck2 targets //... results in shim-related errors.

cc @JakobDegen since you've done (most of?) the work on external cells.

Thanks a lot !

[cormacrelf]

It works with a single file gist. This is with buck2 2024-06-15, I think.

Probably the biggest difference between this and the buck2 repo is that it doesn't have a .buckconfig. That's something external cells are not supposed to support.

; .buckconfig

[cells]
root = .
gist = gist

[external_cells]
gist = git

[external_cell_gist]
git_origin = https://gist.github.com/f7ec39ae8a1003dddf61953b80e06eb3.git
commit_hash = b5cbfc06d0447c4bbadbfad1ca74639c48f19102

[project]
ignore = .git

# BUCK

load("@gist//:rules.bzl", "myrule")
myrule(name = "mine")

buck2 build :mine

[cbarrete]

I think you're right, since I've been able to build the examples in https://github.com/benbrittain/oci-rules (which use external cells) just fine in the meantime. That's a bit of a shame, I guess I'll submodule the buck2 repo for now and build it with reindeer.

Having clearer error messages would be nice, but I'll close this for now as what I'm trying to do is unsupported.

Thanks !

[cormacrelf]

If you must build from source, I recommend building it with Nix. You can also use the buck2 that's on nixpkgs if you are fine with whatever version it happens to be. Otherwise the binaries on GH releases are fine.

In short, the way to build from source is: in a nix flake, using https://crane.dev to build it (which is not the same crane as the one that does OCI stuff), using direnv to load nix when you enter the folder. I believe the buck2 repo has a flake with the build time dependencies listed, and you'll need to use the same nightly version (via oxalica/rust-overlay) as the one in buck's rust-toolchain file. Pass { buildInputs = [...]; BUCK2_BUILD_PROTOC="..."; ... } to crane, and add the output of that to your dev shell.

The main benefit of doing it this way is Nix will cache it, and direnv will switch versions for you if you swap branches. But I would only bother if you need to patch buck2.

[cormacrelf]

Also cargo-based builds of buck2 are meant to be faster because they use jemalloc. Nix+crane does build with cargo.

[cbarrete]

Nix is not an option: I'm trying to build buck2 as a library to build a custom test runner. It needs to be built as part of a buck2-based monorepo, with our own rust toolchain, so as far as I can see, my options are:

• depend on it as an external cell, but this isn't supported due to the .buckconfig issue you pointed out (it may be possible to work around this, but it's not trivial to build buck2 with buck2 right now, even just the CLI, so I don't necessarily want to go down that path)
• use reindeer with a buck2 submodule (since the version on crates.io is not updated), I'm trying this right now and it's not plug and play, but I'm somewhat confident that this can work
• give up and just use cargo, which I really want to avoid

[stepancheg]

cargo-based builds of buck2 are meant to be faster because they use jemalloc

FYI our long term plan for cargo version is to be good enough to compile buck2, but fully featured buck2 can be build only with buck2.

Because supporting every possible non-trivial feature concurrently in cargo build and in buck2 builds is more work.

(We should make buck2-built buck2 use jemalloc.)

[stepancheg]

I'm trying to build buck2 as a library to build a custom test runner

FYI, we plan (but without specific date) is to rewrite buck2 test completely. Internally the project is called "TestInfo V2" — rules will describe how to discover and run tests, not a test runner. External test runner will not exist.

So in your place, for now, until TestInfo V2 is implemented, I would probably avoid implementing test runner (which is nontrivial amount of work and will stop working one day anyway), I would do this instead:

• implement a wrapper for buck2, and called it "buck2"
• in the wrapper, parse command line arguments, and if the command is test, run real-buck2 build (or some fancy BXL, or buck2 run), fetch the binaries, and run them to obtain test results

[cbarrete]

Hmm, that negates a lot of the upsides that we wanted to leverage, including:

• "CI is just running a DAG"
• RTE for free, by default, including not having to setup additional infra (which would be required with your suggestion, and is close to a deal breaker for me)
• Being as close to "standard" buck2 as possible. I don't mind source breakages during updates if I can plug directly into it (of course, a major v1 -> v2 transition might be more painful)

Do you have a timeline for TestInfo V2? Any further information I can read about it (there is only a stub in the docs right now)?

[stepancheg]

Do you have a timeline for TestInfo V2?

We understand this is a critical issue for opensource buck2, but the work was not started. CC @JakobDegen.

Any further information I can read about it (there is only a stub in the docs right now)?

It is in internal docs for now. We plan to publish more such docs externally.

[cbarrete]

Ok, thanks for the clarification! I assume that this also means that you don't have an estimate for when the work is expected to start then, right? I'm just trying to plan accordingly.

[cbarrete]

I'm reopening this with a more accurate title after playing around with this for a few hours.

tl;dr: It seems to be possible to depend on rules (and possibly also macros and plain Starlark functions) from external cells, but not targets, which makes depending on buck2 projects as libraries... challenging.

I have tried several approaches to building against buck2:

• Depend on it as an external cell
• Depend on it as a submodule
• Depend on it as a Cargo crate and use reindeer

The first two seem more or less equivalent, as I hit the same issues and errors with both, which look something like that:

BUILD FAILED
Error running analysis for `root//:main (prelude//platforms:default#904931f735703749)`

Caused by:
    0: Error in configured node dependency, dependency chain follows (-> indicates depends on, ^ indicates same configuration as previous):
              root//:main (prelude//platforms:default#904931f735703749)
           -> buck2//app/buck2_test_runner:buck2_test_runner (^)
           -> root//host_sharing:host_sharing (^)

    1: looking up unconfigured target node `root//host_sharing:host_sharing`
    2: Error loading targets in package `root//host_sharing` for target `root//host_sharing:host_sharing`
    3: package `root//host_sharing:` does not exist
                ^----------------^
           dir `root//host_sharing` does not exist

My understanding is that no matter how I do with the cell/cell aliases configuration, transitive target dependencies will resolve the root cell to my project's root, rather than buck2's root cell. Also, this has required me setting shim = buck-out/v2/external_cells/git/681c0d24a76376c75645f6a9dd633790f5f5910e/shim in my [cells] configuration which is an unfortunate hack which I haven't found how to avoid.

As a sanity check, I did similar tests with https://github.com/benbrittain/oci-rules, which is significantly simpler than buck2. I got the same results: it is fine to depend on rules from an external cell, but attempting to depend on e.g. @oci//tests:app gets me a package 'oci//tests:' does not exist error.

So my more precise question is now: (how) can I depend on _ Buck2 targets_ from a different project? In particular, is it possible to depend on targets from the Buck2 project (which has unconventional cells and uses reindeer for third-party dependencies)?

---

For the record, I have also tried depending on buck2 via reindeer by taking the fixups from the buck2 repo, but the output that I got from it was incomplete (mostly calls to buildscript_run were missing) for some reason. I will likely explore this some more since I can't seem to make native buck2 dependencies work right now.