Hacker News new | past | comments | ask | show | jobs | submit login
Please – A cross-language build system (please.build)
318 points by siliconmountain on Nov 28, 2020 | hide | past | favorite | 248 comments



Please build is really fantastic and I loved using it. If it had better ide integrations I'd recommend it over bazel for oss work. It's extremely fast and we'll documented and the community seems very welcoming.

Glad it's getting some discussion on HN. Can't wait to hear other's experiences.


So what language do you build with it? And why was it better than the traditional build system of that language?


I've used it for Python, C, and some Go. The huge benifit of bazel-like build systems isn't that it's better then a languages build system, it's that it's the best every language build system.

It's consistent between all languages and supports a lot of functionality. For example the proto_library and grpc_library rules make building microservices extremely easy. You can build an entire service in like 3 lines of boilerplate.

It also makes it easy to do cross-language deps. For example a c library included into a Java library that uses native bindings to wrap it.


Plz is great for c++, and has a better workflow (IMO) vs (e.g) Cmake. The skylark-like syntax is easier to reason through than cmake generative expressions. Also the use of “plz run” vs “cmake -g; make; run”


cmake generator only needs to be run once per project. After that you use make or ninja directly, and it regenerates the build files if necessary. I can't say it's a strong argument against cmake.


This added user complexity is exactly the argument for plz, though. One of the ideas touted is (to my mind) tight integration, in that plz knows how to handle everything. Whereas cmake just makes project files and leaves it at that.


Are build tools targeted at end-users? Cmake is better for me, the developer, the person who has to build the software


Sadly this doesn’t always work. I have lost loads of time trying to figure this out then doing a clean rebuild.


I love build tools, and work with them professionally, and this tool seems to be getting one thing quite right: it is not injecting itself into the dependency resolution process. Where I see build tooling fall down is where they try to replace the idiomatic dependency modeling tools that exist in each language.

The build logic and CLI experience looks to be well thought out. I really like the native sandboxing support and the intentional exclusion of the calling environment. There's a lot of polish here.


I read Please website (main page, "Getting Started" and FAQ), and still ain't sure about the benefits.

To the maintainers of Please: please make cases and user stories, also comparisons with other build tools.

Many visitors will jump on each "comparison with tool X" where X=each of the current tools I'm familiar with. Only "Bazel, Buck or Pants" are mentioned. Okay no JVM, looks like Please is a better thought out Blaze/Buck.

Please mention CMake and others, like the C/C++-oriented http://floooh.github.io/fips/ (a high-level build system "similar to Rust’s Cargo or Javascript’s NPM, but for C/C++ projects.").


Agreed - pretty seamless chain from build to execution


It's hard to tell what the value proposition is here, apart from vague hand-waving about parallelization. The quick start is not enough to get started actually building something.

Changing build systems is hard, and getting buy-in from the team even harder - so you need to demonstrate value up front.


Hey, I maintain please and will be the first one to admit we’re not the best at marketing it. We’ve mostly been focused on getting it up to scratch and only recently have we been trying to publicise it.

With that being said I have put a good amount of work into the QuickStart.

The code labs are designed to get you up and started in an inviting way. If you have any specific feedback about your QuickStart experience I’d love to hear it.


As someone who has had to run and maintain build systems for several (sadly far too) large projects (5MLoc+), I want a product that sells me on how easy it is to create _correct_ builds, and how impossible it is to create incorrect ones.

So much wasted time goes into diagnosis of incorrect results from incremental builds that most people who implement CI systems never use incremental builds, and always build from scratch. Developers are far too used to having to do things like 'make clean' because their build didn't work quite right. Efficiencies are then gained by doing things like ccache/Gradle build cache, which trace their dependencies better than most naively written build systems do.

Edit: zig is a beautiful exception (https://ziglang.org/download/0.3.0/release-notes.html#cachin...)


Incremental builds with the said tools (Please, Bazel) should be a non-issue due to emphasis on hermetic build from these tools.


Unfortunately, reality is messier than this. For example, Please currently has escape hatches that can be used that nullify these guarantees. What I have found is that if such a mechanism exists it will eventually be abused.

My perspective here is specifically from that of a maintainer of large builds where the code base is always under active development. Murphy's law becomes your enemy at scale.

This is not a criticism of Please - just an observation that as an industry we are not there yet. But, I'm happy to see things moving in the right direction.


I haven't had many issues with non-determinism when using Bazel/Blaze? Especially when you use remote execution or sandboxing it's pretty hard to avoid being hermetic.


Here's what I mean:

    genrule(name = "oops", cmd = "date > $@", outs = ["oops.txt"])
This is admittedly contrived, but sadly not unrealistic.

Again, I am not picking on Bazel here. Correct builds are difficult enough to get right even when everyone involved wants to keep the build correct! But, the reality is that a lot of real-world build conditions can be downright hostile. (did you know that ext4 vs xfs can change the order JARs in the same directory are loaded off the classpath? I wish I didn't!)


Yeah fair point. At Google a lot of these non-deterministic rules are either banned through "date" just not existing on remote execution machines, or through determinism tests that ensure rules behave. Maybe in real-life scenarios it's not as easy as I claim, as you suggest.


For such a system it is only reasonable to have such an escape hatch but maybe a global config forbiding its use would solve your problem ?


Agree with the OP. I went through both the Python and Go quickstarts. They were easy to follow, but I’m also not seeing the value proposition. The build system has different DSLs for different languages so I’m not seeing the advantage of switching from I.e. go install ./... for Go and setuptools for Python? No offense, but you didn’t answer the question: what’s it good for?


In that case, you can stitch both Python and Go builds together in the same interface, and then tie them into further actions (e.g. Docker builds, Kubernetes), again using the same interface. Neither setuptools nor go build are particularly helpful to that goal.

(I'm one of the original developers, but these days do less maintaining of it than tatskaari does)


Thank you for making this!

My rubric for documentation is "how many clicks to code". I found the installation instructions, and started to look for usage, but didn't find it. Too many clicks. That's where I'd start.

PS - IMO the gold standard for this evaluation method is Sinatra.rb, which is zero clicks to code.


Are you referring to this [1]? I had to click Documentation to get here and still didn’t see any code...

[1] http://sinatrarb.com/documentation.html


I think he was referring to the Intro page [1][2].

[1] http://sinatrarb.com/intro.html [2] https://github.com/sinatra/sinatra#readme


I was referring to the landing page: http://sinatrarb.com/


Oh, sorry, that didn't show up on mobile. I understand now, thanks.


What was/is the motivation for this build system that an existing one doesn't satisfy?


We wanted something like Blaze (Google's system); at the time we were using Buck which didn't satisfy us (e.g. only one output per genrule, no directories; and it was more or less impossible to write first-class support for a new language without modifying the core system). Subsequently Bazel got released but we still find that a bit lacking in some areas; e.g. the CLI, the JVM reliance and some of Starlark (e.g. it maintains Python 2 syntax compatibility so can't have type annotations).

We'd also tried Gradle previously but that was pretty awful for anything non-Java which made it a non-starter.

To be clear, I'm one of the original implementors, although I imagine that was obvious already...


fyi, Starlark dropped compatibility with Python 2 two years ago (https://github.com/bazelbuild/starlark/issues/27)

People interested in type annotations can check this discussion: https://github.com/bazelbuild/starlark/issues/106

For most purposes, Bazel dependency on Java is an implementation detail (users don't need to install a JVM), although you might notice it if you need to bootstrap Bazel.

(I co-designed Starlark and I used to work on Bazel)


I’m not sure if the feature was added after you started building Please but Buck does support multiple outputs from a genrule now. It’s kind of hacky though; you write a genrule that produces a directory containing said multiple outputs, and then you can write rules to extract the subcomponents. (I don’t know why I feel compelled to point this out. I worked on Buck for about a year, several years back. )


At first this wasn't a singular clear answer I was hoping for, but the many points being made here make it clear that it's a complex area and there's room for another advanced/high-perf build system to cover a broader range of usage.


This makes me sad.

You hopefully have tried out many other build systems by now, been frusted with all of them, and only then made your own. And with all that experience with the status quo, marketing should be easy.

If you haven't done that, this project is a net-negative on the world, because the proliferation of build systems exacerbates Conway's law and balkanizes our software commons.

NIH in FOSS is not free.


I am not an author, but was around when we decided to create Please in 2015.

When we created Please we were currently using Buck from Facebook. As has been covered elsewhere at the time it had some limitations which meant that it was getting more and more expensive to workaround (no multiple outs from rules so doing things like sourcemaps was hard etc). We had previously migrated from a set of disparate build tools (Gradle, Gulp etc) to Buck after evaluating it against Pants (we actually trialled our repo with both).

We all had experience with Blaze (Bazel did not exist in 2014/2015) and wanted to get closer to the experience we had in Google, hence the final decision to build our own. At the time we rolled it out we had full Buck compatibility (and actually most of our engineers did not notice initially). Once we were happy it worked, we migrated in full to Please.

Overall it allowed us to provide much better tooling for our developers (queries on the build graph for smaller CI/CD footprints, coverage in all languages, and for rules that can output multiple languages (like protocol buffers) we only build variants in the requested languages rather than all languages).

If Bazel had been open-sourced and easy to extend at the time we would certainly have looked to adopt and improve it (as we did with Buck at the time). But by the time they open-sourced it the two system had diverged in their approach and use cases.

I (obviously somewhat biased) think having multiple principled build systems which prioritise the needs of different communities is good for the ecosystem overall (and allows the sharing of good ideas such as the Remote Execution Interface).


Marketing can be hard not because you don't have something worth marketing, but because marketing is a skill-set largely orthogonal to software development.


That's only true if you define software development in a pretty particular way.

For me, writing code is a way of solving problems for people. That involves listening to them to see how I can help. Then collaborating with them to iteratively make something that fits their needs. Which includes helping them adopt it and get productive with it. To me, all of that is part of software development.

And that's most of the "marketing" that's needed for a free, open-source tool. When that's lacking, I have to wonder to what extent the tool was actually made to solve somebody's problems. As opposed to being created because the developer was excited to build a thing as a technical exercise. Either path is fine, of course, but it's a mistake to confuse the two.


To me, writing code is very much the same way - it is a human endeavor. But, over the years, I have learned that just not all people are the same way, and that's okay, too.

Growing up, one of the jobs I took was as a system's administrator at the same company my father worked at. It was a small company, maybe six people or so, doing contract work for aerospace. One of the older men who worked there held his Ph. D. in aerospace engineering. He was kind, warm and brilliant... but he struggled with things like sending coherent emails to clients. But that was okay - other people were able to take that on.


That's totally fair. But...going on a hunch, such a person would also make a crude 90s-style website.

Whoever made this clearly knows about slick presentation --- they've got all those colors and shapes pizzas --- so they either capable with slick communications, or work with someone who is. My patience for lack of explanation is therefore a lot thinner.


I'm talking about "marketting" to other software developers, not to non-software developers. Communication with people like oneself is a core skill, not a specialized trade.

This isn't a paid product, after all.

(I'm also a traditionalist that thinks writing good prose and writing good programs are related skills.)


> marketing should be easy

I’m sorry, what?!

Do you realize the “sell something to people” industry has been at least as big as the tech industry for many decades and is the one growing faster and faster?

Also marketing has historically been a problem for people who make stuff.

Also, better have the Balkans of software than the USSR of mega corps.


You should also say upfront that it doesn't support Windows.


Thanks for all the hard work! I'm very excited to see where this goes.

Have you considered sponsoring, or accepting donations, to work on IDE integration? That's pretty much the only reason I migrated my current employers codebase over to Bazel instead of Please.


Hey! Glad you liked it :D Please is owned by Thought Machine. They’ve recently (8 months) brought me on the project full time. We’re planning on putting together a dedicated IDE team together but no promises.


As a big fan of Bazel (ok, Blaze), I'm curious what's different than what Blaze offers


Bazel has a lot of things "missing" that had to be cut out from the OSS release. Things like grpc rule sets are extremely fragmented right now. It's a bit of A saying "the B team should maintain the rules" and B saying "the A team should maintain the rules". As such almost every single language has inconsistent support for everything: code coverage, protobuf, grpc, external dependency management, IDE integration. Also because there's no stubby, sponge, or other cool internal tools a lot of the benefits from blaze as a system are not externalized.

Also, the docker support in Bazel is pretty "different". It's not bad, but it's not immediately obvious to newcomers how this is meant to work.

It's getting better every day though.


A few sample build configs of varying complexity would be nice to see at a glance to understand what you are getting into without digging through the weeds.


Typically (general purpose) build systems make you choose between reproducibility and performance. Please and other related tools aim to change that (however, in my experience they all fail for one reason or another—not because the problem is inherently intractable but rather because these tools are underinvested-in such that they’re infeasible for smaller organizations to operate and larger organizations can afford to roll their own in-house and get the benefits of a tool suited to their specific needs).


I think this “missing middle” is a common pattern in expert tools. The economics are challenging because people quickly move through the skill level where the tools are most useful.


The FAQ suggests it's inspired by Google's blaze, because bazel hadn't been open-sourced at the time of writing.

https://please.build/faq.html


For me it has the feeling of something I would play around with in my free time. But as a drop in replacement I agree, it's a bit of a hard sell for me.


Yup fair enough. Migrating build systems is hard. I've put some work into making migrating from a `go build` project to Please a bit easier. Banzai cloud recently migrated their [Pipeline](https://github.com/banzaicloud/pipeline) project over to use Please but admittedly it was quite a lot of work for them.


If you have used (e.g) bazel before, I strongly encourage you to give it a bash, very easy to pickup and “feels good” to use


How easy is it to "port" BUILD files between the various open implementations (Bazel, Buck, Pants, Please)? If you have a project someone else wrote with the intention of using it in e.g. Bazel, can you integrate it into your Please build system, or would you need to treat it as an opaque build process (same as if you were shelling out to ./configure && make)?

As a concrete example, one frustration at $DAYJOB is that building certain Google-released OSS is hard to integrate with our existing build system because Bazel is pretty opinionated about compiler-provided headers, output paths, etc. We're calling Bazel as a subprocess during our build and making various files available to it to make our compiler build visible to it, and then we pack up the results. I'm pretty sure if we somehow adopted Bazel for our entire build this would be much better - would it also be better if we adopted one of those other systems?

(Maybe one way of asking this is, are any of the corporate sponsors of Buck, Pants, or Please building third-party Bazel code from source? Or vice versa?)

A broader question: would it be worth defining a compatible subset of BUILD file syntax and library calls (i.e., not just Starlark but also the rules themselves for common things like building C libraries or JARs)?

Are we in the build-system equivalent of the vendor-specific C / C++ / UNIX implementations, and will a cross-vendor standardization effort emerge one day?


I used to work on Bazel and Starlark. In the past, I talked with engineers working on other build systems to aim for some compatibility. The was some interest (even Microsoft people were interested to support Starlark rules in their tool), but it didn't work.

On the BUILD file level, Buck is using the Starlark. Pants and Please are close enough that some tooling (e.g. buildifier) can be shared. But that's about it.


I've only seriously worked with bazel, but it seems pants and bazel have at least enough compatibility for twitter to migrate to bazel. They gave a talk[0] at bazelcon this year about the effort.

[0] https://opensourcelive.withgoogle.com/events/bazelcon2020/wa...


For the first part of your question - it is not 1:1. There is some limited capacity for bazel integration, but massaging is needed (IME) for protos, nvcc rules & so on


Style issue: the left nav text often overflows into the content area on mobile, making it painful to read.


They make some lofty claims about Python packaging, has anyone used this tool in a reasonably high stakes environment for Python? How was the experience?


I come from a sofware world not typical for HN. I am a (very) low level software and FPGA guy. Despite my best efforts, I don't understand: What do any of these tools do that Make does not? Are they faster and easier to use? Do they work better?


Makefiles operate at a low level of abstraction. The Makefile abstraction is "run this command when the output files don't exist or are older than the input files." You manually specify every tool to be run and all of its arguments.

These tools operate at a higher level of abstraction. The BUILD file abstraction is something like "I am declaring a C library with these sources and headers." The build system then turns that into a set of commands to run. This higher-level abstraction provides a bunch of compelling features:

- Reliability: it's easy to write Makefiles that are fragile, for example one that works when run on a single CPU but fails when run with -j, because of undeclared dependencies. When the tool is in charge, it has more control and can sandbox each build step so that undeclared dependencies simply won't be available in the sandbox.

- Reproducibility: similar to the previous point, it's easy to write a Makefile that produces incorrect results with an incremental rebuild. For example, unless you implement your own header scanning (or manually list your header dependencies) "make" won't know to rebuild a source file when a header changes. These tools can ensure that builds are reproducible and incremental rebuilds are always correct.

Besides that, working at a higher level generally means writing less code to accomplish the same task, and the code you do write is closer to expressing your actual intent compared with a Makefile.


> The Makefile abstraction is "run this command when the output files don't exist or are older than the input files." You manually specify every tool to be run and all of its arguments.

> The BUILD file abstraction is something like "I am declaring a C library with these sources and headers."

This is wrong. Even ninja has generic rules. Here's an example of a minimal makefile:

  OBJ = src/a.o src/b.o src/c.o
  libfoo.so: $(OBJ)
          $(CC) -shared -o libfoo.so $(OBJ)
> When the tool is in charge, it has more control and can sandbox each build step so that undeclared dependencies simply won't be available in the sandbox.

There's no reason this shouldn't be possible with make; it just hasn't been implemented so. Do bazel/buck/please actually do this? As far as I know tup is the only tool that actually verifies inputs/outputs of rules, and it needs FUSE to do so.

> For example, unless you implement your own header scanning (or manually list your header dependencies) "make" won't know to rebuild a source file when a header changes.

At least with GNU make, it's very easy:

  CFLAGS += -MMD
  -include $(patsubst %.o,%.d,$(OBJ))
True, it's a bit of a footgun, but by no means difficult.

Make has problems, but the ones you listed aren't they.


> Here's an example of a minimal makefile:

Your example does not contradict what I wrote. You manually specified the tool to be run ($CC) and all of the arguments to that tool.

It's true that there is a level of indirection through the $CC variable, but you're still operating at the level of specifying a tool's command-line.

> There's no reason this shouldn't be possible with make; it just hasn't been implemented so.

Make is 44 years old. If it were an easy extension to the existing paradigm, there has been ample time to implement such an extension.

> Do bazel/buck/please actually do this? As far as I know tup is the only tool that actually verifies inputs/outputs of rules, and it needs FUSE to do so.

Bazel certainly has some amount of sandboxing, though I don't think it's quite as complete as what is available internally at Google with Blaze. I haven't used Buck or Please, so I can't speak for them.

> True, it's a bit of a footgun, but by no means difficult.

Well footguns aren't great. :) As just one example, any header that is conditionally included (behind an #ifdef) could cause this cache to be invalidated when CFLAGS change, but Make has no idea of this.


> You manually specified the tool to be run ($CC) and all of the arguments to that tool

Notably, I didn't specify the arguments to be used to compile the source files. But if you insist:

  linklibrary = $(CC) -shared -o $1 $2

  libfoo.so: $(OBJ)
          $(call linklibrary,$@,$(OBJ))
Notably, linklibrary can be defined according the platform, or according to dynamically set variables, giving you the same level of flexibility as make.

>> There's no reason [sandboxing] shouldn't be possible with make; it just hasn't been implemented so.

> Make is 44 years old. If it were an easy extension to the existing paradigm, there has been ample time to implement such an extension.

‘Nobody's done it yet, ergo it's not possible or easy’ is not a valid argument.

> just one example, any header that is conditionally included (behind an #ifdef) could cause this cache to be invalidated when CFLAGS change, but Make has no idea of this.

Then you specify that the source files depend on the makefile.


> Then you specify that the source files depend on the makefile.

Then you gratuitously rebuild everything whenever the Makefile changes, even if you only changed a comment.

Also this scheme is incorrect if the Makefile was written to allow command-line overrides of variables like CFLAGS, as many Makefiles do.

But these are just details. The larger point is this. The language of Bazel is defined such that builds are automatically correct and reproducible.

While it's true that Make has some facilities like "call" that support some amount of abstraction, it is up to you to ensure the correctness. If you get it wrong, your builds are back to being non-reproducible.

It's like the difference between programming in a memory safe vs a memory unsafe language. Sure, every thing you can do in the memory safe language can be done in the unsafe language. But the unsafe language has far more footguns and requires more diligence from the programmer to get reasonable results.


And now we are in agreement :)


> ‘Nobody's done it yet, ergo it's not possible or easy’ is not a valid argument.

Ultimately, make and co are text oriented, while bazel and co are object oriented. Hacking object oriented capabilities into a text-oriented language isn't particularly fruitful or ergonomic.


I don't follow.

Bazel and make are both text-based languages for describing symbolic, abstract structures. Just like pretty much every other programming language.

Bazel and make both use the same abstract structure, just like pretty much every other build system: a directed graph of build directives and dependencies.


Fundamentally, bazel and make treat "targets" differently. A make target is an invokable thing. That's about the extent of it. You have a dag of invokables, and invoking one will cause you to invoke all of its dependencies (usually, other people have discussed the limitations of make's caching already).

But let's look at how a rule is implemented in bazel[0]. Here's a rule "implementation" for a simple executable rule[1]:

    def _impl(ctx):
        # The list of arguments we pass to the script.
        args = [ctx.outputs.out.path] + [f.path for f in ctx.files.chunks]

        # Action to call the script.
        # actions.run will call "executable" with 
        # "arguments", saving the result to "output"
        # access to files not listed in "inputs" will
        # cause errors.
        ctx.actions.run(
            inputs = ctx.files.chunks,
            outputs = [ctx.outputs.out],
            arguments = args,
            progress_message = "Merging into %s" % ctx.outputs.out.short_path,
            executable = ctx.executable.merge_tool,
        )

    concat = rule(
        implementation = _impl,
        attrs = {
            "chunks": attr.label_list(allow_files = True),
            "out": attr.output(mandatory = True),
            "merge_tool": attr.label(
                executable = True,
                cfg = "exec",
                allow_files = True,
                default = Label("//actions_run:merge"),
            ),
        },
    )
This is, admittedly, not easy to follow at first glance. Concat defines a "rule" (just like cc_binary) that takes three arguments: "chunks", "out", and "merge_tool" (and "name", because every target needs a name).

Targets of this form have metadata, they have input and output files that are known and can be queried as part of the dag. Other types of rules can be tagged as test or executable, so that `blaze test` and `blaze run` can autodiscover test and executable targets. This metadata can also be used by other rules[2], so that a lot of static analysis can be done as a part of the dag creation, without even building the binary. To give an example, a rule like

    does_not_depend_on(
       name = "check_deps",
       src = ":opensource_thing",
       forbidden_deps = [
           "//super/secret:sauce",
       ]
    )
can be built and implemented natively within bazel by analyzing the dependency graph, so this test could actually run and fail before any code is compiled (in practice there are lots of more useful, although less straightforward to explain, uses for this kind of feature).

Potentially, one could create shadow rules that do all of these things, but you'd need to do very, very silly things like, off the top of my head, creating a shadow filesystem that keeps a file-per-make-target that can be used to query for dependency information (make suggests something similar for per-file dependencies[3], but bazel allows for much more complex querying). That's what I mean by "object-oriented". Targets in bazel and similar are more than just an executable statement with file dependencies. They're complex, user-defined structs.

This object-oriented nature is also what allows querying (blaze query/cquery/aquery), which are often quite useful for various sort of things like dead or unusued code detection or refactoring (you can reverse dependency query a library that defines an API, see all direct users and then be sure that they have all migrated to a new version). My personal favorite from some work I did over the past year or so was is `query --output=build record_rule_instantiation_callstack`, which provides a stacktrace of any intermediate startlark macros. Very useful when tracking down macros that conditionally set flags, but you don't know why, and a level of introspection, transparency, and debugability that just isn't feasible in make.

That's what I mean by object-oriented vs. text oriented. Bazel has structs with metadata and abstractions and functions that can be composed along and provide shared, well known interfaces. Make has text substitution and files. While a sufficiently motivated individual could probably come up with something in make that approximates many of the features bazel natively provides, I'm confident they couldn't provide all of them, and I'm confident it wouldn't be pretty or ergonomic.

[0]: https://docs.bazel.build/versions/master/skylark/rules.html

[1]: https://github.com/bazelbuild/examples/blob/master/rules/act...

[2]: https://docs.bazel.build/versions/master/skylark/aspects.htm...

[3]: https://www.gnu.org/software/make/manual/html_node/Automatic...


‘same level of flexibility as make’ should read ‘same level of flexibility as bazel


FUSE, strace, and namespacing were all the mechanisms I found. Bazel uses separate wrapper program which you could reuse in other build systems, so there is no fundamental problem with adding a "hermetic builds" feature to other build systems like Meson or Cmake.

http://beza1e1.tuxen.de/hermetic_builds.html


Please takes sandboxing a bit further using kernel name-spacing to isolate builds and tests. It's an opt-in feature but you can bind to port 8080 in your tests and run them in parallel if you do ;)


This was an extremely good explanation for a non-trivial subject. Props for written communication ability.


Thanks, I appreciate it!


Make has a laundry list of problems:

- Cannot handle multiple outputs for a single rule

- Does not rebuild when flags change

- Make rules may contain implicit dependencies

- Slow for large codebases

- Does not understand how to build for multiple platforms, sucks for cross-compiling

- Recursive make sucks (job control does not work across recursive invocation boundaries)

- You must create output directories yourself

- You must create your own rule to clean

This adds up to a fragile & slow build system, where you have to do a full rebuild to have a reasonable level of assurance that your build is correct—incremental builds aren’t safe.

There have been a few attempts to make a “better make” over the years like Jam, SCons, Ninja, Tup, etc. each with their own pros and cons.

The new generation of build tools—Bazel, Buck, Pants, and Please are all an attempt to redesign build systems so that the design is resistant to the flaws that plague Make build systems. You can use a shared cache (shared between different users) fairly easily, and you have reasonable assurance than incremental builds are identical to full builds (so your CI pipeline can move a lot faster, developers almost never have to "make clean" when they change things, etc).

Personally I’m working on a project right now that uses Bazel (which is similar to Please) and is for an embedded system. It’s been a great experience, and I can pass a flag to Bazel to tell it to build for the embedded target using the cross compiler or for the native host--that makes it easy to share code between tools and the target system, and I can do things like write tests that run on both the target & host. Anyone who does any cross-compiling is missing out if they are using Make—but, do note that setting up a cross-compiling toolchain in Bazel isn’t exactly a cakewalk.


Make does support multiple outputs, though the syntax sucks. Most of what you are annoyed with though is like being annoyed at C for the same reasons: Make is a programming language with a built-in dependency mechanism, and as such you can use it to build whatever you want... now, does it already come with whatever you want? No. I can appreciate wanting something which does. But such systems usually then give you what they want. I don't want my programming language to solve half of these things you want, and yet somehow I have built things on top of Make that solve even the problem of building for arbitrary embedded cross compile targets. (And hell: of cross compile toolchains is what you care most about, the king of that space is autotools, which not only correctly supports compiling to everything always every time out of the box, but somehow manages to make it possible with nothing more than you passing some toolchain configuration as command line arguments to the generated configure script... and one might note that it generates Make, though without taking much advantage of what makes Make great.)


> Make does support multiple outputs, though the syntax sucks.

No, it doesn’t. There’s NO syntax for multiple outputs. If you can show me what the syntax is and prove me wrong, I’d love to see it. At best, there are workarounds for the lack of multiple output support, with various tradeoffs.

> Make is a programming language with a built-in dependency mechanism, and as such you can use it to build whatever you want...

Make is NOT a programming language. End of story. You can… kind of… build things with it, given that variables in Make can be used like functions, but it’s goddamn horrible and you shouldn’t do it because nobody will want to use it and nobody will maintain it.

At most, you can build something on top of Make, but you’re still facing the limitations of Make and working around them. If you are interested in building something on top of a low-level build system, you want Ninja instead of Make, because Ninja is good at that. Make isn’t.

Make is, at best, something you would want to use for a small C program where all the files are in one directory. Once you grow past that use case, my rule is that you shouldn't be using Make any more, because there are just too many things that Make does wrong.

Make is successful because eh, it’s good enough for small things, you can suffer through the pain if you need to use it for big things, and it was the first tool to solve this problem. We have better tools now. We had better have better tools now! Make is in its fifth decade… if we didn’t improve on Make in that many years, that would be a damning indictment of the entire field.


GNU Make does support multiple outputs, but the feature is very new (it's in the latest release that came out earlier this year), so if you didn't happen to catch that release announcement you probably missed it. The support is called 'grouped targets', documented in the second half of this page: https://www.gnu.org/software/make/manual/html_node/Multiple-... -- the syntax has &: in the rule line.

(One point you don't mention in your list of reasons why Make is successful is that it's reliably available everywhere. For projects that ship as source for others to build, that matters, and it delays uptake of fancy new build systems in that segment.)


The big advantage of make to me is I understand it and can figure out what happens when things go wrong. When something doesn't work the way I want with cmake or autotools (I haven't used Bazel etc.), I have to randomly start googling things. Sometimes I literally resort to editing the generated cmake Makefiles because I have no idea how to tell cmake what I want it to do...


I am interested in setting up cross compiling environment with Bazel. Do you have any recommended documentation that I can read?


The documentation is not so great. What I’m doing is enabling platforms (https://docs.bazel.build/versions/master/platforms-intro.htm...), defining the OS and CPU target for my system, copying the configuration out of Bazel’s “local_config_cc”, and modifying it to fit my use case. This didn’t take me very long, but it’s also not the first time I’ve done it.


Thanks! I agree that the documentation isn’t that great and I feel there are lots of holes.


Did some searching and found this documentation on customizing C++ toolchain. I'll give it a try.

https://docs.bazel.build/versions/master/tutorial/cc-toolcha...


> Make has a laundry list of problems

So do C, and Bourne shell. And SQL.

And yet, here we are. Are the benefits enough to overcome incumbency?


Make is only good at one thing: calculating how to execute a dependency graph to produce a specific target.

Everything else that is needed for a modern build system is lacking:

    - expressive scripting language

    - platform-related configuration

    - feature-related configuration

    - handling hierarchical projects

    - properly handling rebuilds when the Makefile itself has been changed

    - dealing with dynamically generated parts of the dependency graph

    - handling third-party dependencies

    - build reproducibility

    - etc... the list is very long
This is why people these days mostly take the path of generating Makefiles, because make is only good at one thing: executing a statically defined dependency graph.


Make is a programming language: it is only good at executing that language, and that language kind of sucks, but saying Make is somehow only good at executing static dependency graphs is a fundamental misunderstanding of Make (but AFAIK might be a good description of say, ninja). All of the Make scripts I have written for the past 15 years have dynamically generated configuration and autodetected files and calculated version numbers and essentially done all of this work that somehow you think Make can't do.


Anything that is (like make) turing complete can be made to do all the things you list.

At a very minor cost after all: your sanity.


I think there's a few advantages of bazel-like systems:

1. Reproducible/Hermetic: your builds cannot, say, access the network during a step or produce changing binaries for the same input. This makes sure that once you've downloaded your deps locally everything builds correctly from then on.

2. Caching: Because everything is reproducible in bazel-likes everything is cachable. Dependencies, builds, and even tests! If you've worked with Makefiles you've likely run `make clean; make tests` or something similar. I've never needed to do `bazel clean` to make sure things were working right.

3. Visibility: you don't only control who can see what in your source code. Different dependencies can be marked as private/protected/public to control who can import them. This is a huge boon to large monorepos.

4. Everything is uniform: code generation, compilation, etc is all described in "rules" and "toolchains" and can "easily" expanded to other languages. The community manages a Rust and Golang rules set for bazel itself and they're better then the Google rule sets for Java (only "officially" supported rule set right now) in some areas.

So if you have a lot of code/internal libraries/code generation, what to write a LOT of unit tests and cache their results, and write code in multiple languages bazel is probably for you.

You can also use tools like BuildBuddy [0] to share a cache with your team and extract build performance metrics to identify areas you can improve performance in.

[0] - https://www.buildbuddy.io/ or https://github.com/buildbuddy-io/buildbuddy


The problem with Make is that it’s never just Make. It’s also Autoconf, Automake, m4, and god-knows-what-else. If Make is so great, why are people resorting to generating the Makefiles? There’s clearly some kind of deep UX problem there.


> Autoconf, Automake, m4

all of which are even worse than make itself.


I think it’s 1. people find makefiles confusing, and if someone’s introduction to the concept is a large project they may never pick up the basics and 2. Make’s fundamental assumptions: that each command creates a single, sensible artifact that is the input for the next phase, maps poorly to the dynamic build processes of browser and phone ecosystems, which are themselves barely controlled explosions of complexity


Make build configurations can be difficult to understand for newcomers in the industry. If the goal is to obscure the code, by all means continue using the older tools.

If the goal is continued maintenance, then encouraging new engineers to explore and read the codebase with tools they can comprehend is critical.

disclaimer: have not used these particular tools, but the domain is nice and polite


I have use all of these tools and your take isn't accurate at all. Make doesn't obfuscate anything more than Plz or CMake or whatever.

Here are some real reasons why Make isn't the end-all:

- Make doesn't allow platform selection in a nice way. - Make doesn't work on Windows natively (no, NMake doesn't count). - Recursive make doesn't work well at all. - Make doesn't track byproducts or deleted artifacts. - Make doesn't have auto-clean. - Make doesn't facilitate out-of-source builds. - Make itself isn't a scripting language (arguably good/bad). - Make doesn't facilitate compiler selection cross-platform, so things like MSVC are nearly impossible to integrate nicely without the use of the developer tools prompt. - Make can't (elegantly) handle a number of rule cases, such as single-input multiple-output rules (it can, but it's a hack).

Not sure why you think Make is unapproachable. That's like saying shell scripting is unapproachable. No developer worth a damn is going to avoid shell scripting, and as long as they understand "this file turns into this file using this command" then Make makes sense.

As with all tools, Make and Autotools and CMake and probably Plz will be misused by developers that think they're being clever. In actuality, they make things worse at best and unusable/unmaintainable at worst.

As a build systems designer, I've evaluated Plz personally and found it to be neat but not suitable for most of the problems I think build systems should solve.


Re win32 I've had not-terrible experiences with dmake.


What tool would you pick over plz?


CMake still. I know there is something that will come out some day that is better, but until that day, CMake is still the best.


The short answer is yes, they are (well, can be) easier and faster and work better.

I would strongly recommend watching this talk if you want to understand why systems like Bazel and Please were created:

Build Systems with Andrey Mokhov (Jane Street)

https://www.youtube.com/watch?v=V9YA32uV3Ls


I think one good way to think about it is all of these build systems create Make environments. They create the makefiles and have all the other tools that go with it.

So at the end of the day you end up with an executable artifact, but hopefully you did a lot less work to get there.


Only garbage like autotools or CMake generates Makefiles, all remotely sane build systems (including the various bazel/blaze derivatives) do away with make, which even sucks as a low level abstraction.


I don’t about these new build tools, but I’ve usually found ninja to be faster than make in most situations at least.

Also curious to know, is there no work done in the realm of developer productivity inside the FPGA world?


No unfortunately. The FPGA "build" realm is mostly a clusterfuck of proprietary tooling. It all has poor to nonexistent integration with, well, anything else. There are some things that are okay for simulation like Cocotb, but for actually building hardware FPGA designs, you are locked into the vendor's tools. The only exception is if you use a small, slow, and outdated FPGA, not suitable for large designs used in industry. For these reasons, there isn't much of an open source community making things like Please and Bazel specific to FPGAs. You can set it up yourself, but a packaged product that just works out of the box doesn't exist to my knowledge.


I work in a similar field and this is accurate for us too, except our tools do at least have some integration with Make (you can generate makefiles from the proprietary project files, although it's not useful to us and we don't use it). Like most (I assume), we've built our own build language/system on top of Make. It works but only in a few well-defined ways that modern software shops would probably baulk at. We don't support parallel builds, no caching, only coarse grained incremental builds (at a subproject/library level) and who knows what else that others benefit from.

I've only looked at modern tools briefly but everytime I come away realising we'd have to re-write everything. There is no out-of-the-box support for our tools/language and it would be a lot of work to learn someone else's DSL, re-implement everything and then potentially discover a load of ways it's broken for our workflow. Personally I think it'd be worth it in the long run but it's a really hard sell.


Make is not written in Golang/Rust/Haskell and lacks a cool logo and hip landing page.


I can imagine a project that successfully replicates the entire google developer environment (distributed build system w/ caching, monorepo with presubmit checks, code review, testing, etc) would be successful since ex-googlers would be likely to advocate for it inside their own organizations and most orgs don't have the engineering time to build all this themselves.

Without this tooling large organizations tend to silo which creates high engineering costs (every project starts from scratch, no shared frameworks/libraries, no standardizing around best practices, etc).


A surprisingly hard sell even if your organization has a supermajority of ex-Googlers. There's a really strong meme out there that if you are smaller than Google, which is almost universally true, then you shouldn't adopt their practices even when they are zero-cost and obviously superior.


For me the "obvious" part of "obviously superior" is worth interrogating. Googlers have a very particular way of thinking of things. Which works well enough for Google, I guess. But there are a lot of ways to to good work, and not all of them are compatible.


Sure, I think thought needs to be put into any development decision. I think a new project should carefully consider, for at least several hours of research and discussion, what revision control system to use, instead of just using git without consideration of the consequences.

There are already book-length arguments that I like to refer to for specific topics, such as trunk-based development.

https://trunkbaseddevelopment.com/


I agree. A big missing link at my organization is a VCS that can make it easier to checkout and operate on only a subset of a large (many GBs) monorepo.

Unfortunately, microsoft had to move away from VFS for Git to some a more limited-in-scope approach: https://github.com/microsoft/scalar


Silo’d development isn’t a matter of tooling, it’s a matter of communication.


There's an interplay between them. Communication always has a cost; there are certainly things you can do to the constant factors, but in a 100-person engineering organization with 20 teams, it's fundamentally going to be much easier for an engineer on one team to write a build system (or pre-merge CI flow, or whatever) that meets the needs of their four teammates working on the same code than one that meets the needs of all 20 teams.

The advantage of a running, pre-existing system (and the theoretical advantage of an off-the-shelf system like this that you can supposedly turn into a running system) is that it's been built to satisfy most of the needs of a wide variety of developers working on all sorts of things, and so each engineer on each of those teams that says "All right, I'll spend some time thinking about how we build and release code" is incentivized to start from the common starting point, even if they have to customize it. That means that if two or three teams end up working on similar-enough projects down the line, the shared tooling can actually support communication between the teams. Without the shared tooling, they can communicate all they want, but it won't be rational for any team to abandon its existing tooling. (And of course you need the communication too - both are required.)


I hear you. I do. How do you get 20 teams working on disparate codebases, and probably languages, to share their knowledge across teams? How does a team working on Java benefit from something someone found in Django from another side of the company? Organizations need standards. Developers require it. We want the thing we are programming against to not change right? We want our codebase to build when we say build. All tests green. Happy. Productive. Engineering leadership needs to organize standards around engineering culture and tooling, it could be championed by a team that did it, or by research. It doesn't matter. What matters is that leadership facilitates communication into what is working, what isn't, why, and what can we do about it?

At the micro-level, tools help, no doubt about it. At an org level, communication helps (documentation is a form...). Even if using different tooling, concepts and strategies are the same. This isn't to take away from the argument that this tool the OP posted is pretty awesome.

If you want change, be the change. Evangelize the tool through your org, not just your team. If your org is silo'd, break those barriers down with a friendly lunch-n-learn or lightning talks session. We tout empathy for our customers, how about empathy for our coworkers too? Others may have the same pain...


And tooling. If tooling is consistent engineers feel much more confident crossing borders. No one speaks up or contributes without confidence.


First thing every software engineer does after leaving Google is rewrite Blaze. Just like every site reliability engineer rewrites borgmon. What I chose to do myself was make Blaze happen using a Makefile config. There's a blog post somewhere where Google talks about why they switched from GNU Make to Blaze c. 2006. if I remember correctly it basically boiled down to not having strict dependency checking. So I thought, why not simply avoid their published mistakes without changing build systems? I learned everything I could about make, discovered that variables are secretly lambdas, and that enabled me to figure out a way to have a static archive package for each folder, which also avoided the need to write a makefile code generator. Then I wrote a simple C program to check incremental elf symbols so the build graph stays sanely formed. It totally got make working like a dream in a way I simply hadn't seen before. https://github.com/jart/cosmopolitan/blob/master/tool/build/...


Reading your words, it looks like you have made a kind of make replacement that just works. Following your link, it looks like... what ? I don't know, some tool that checks some build configuration, augments C and postprocesses executable files?

(Looking at https://github.com/jart/cosmopolitan it's an interesting hack. Might be useful in some cases, for example command-line tools like git.)

Have you made something that looks like a better make? If so, it would be nice to know more. Thanks.


Sorry here's what a build file looks like, which should hopefully clarify: https://github.com/jart/cosmopolitan/blob/master/libc/mem/me... As you can see, it's just vanilla make. The package program is just a helper that checks to make sure DIRECTDEPS variables are correct. That's the part Google said was tricky back in 2006. Have you ever seen people talk online about how it's super difficult to have a single makefile build multiple library files? Particularly static ones? Like the pitfalls of making sure the linker arguments are the correct order, or removing a dependency from one will break all the others that needed it transitively? This is how it can be done. I found out it just requires following a highly specific convention, where it's a little toilsome to type, but so rewarding once it's done. It also means I don't need to ask users to install a cutting edge build system to build my code, since I've just seen too much drama in that space.


Every ex-Google SRE I know is glad they don’t have to deal with Borgmon anymore. I don't think it’s popular these days.


All the more reason to invent something that improves upon it, since love it or not, it's a must have tool.


A Google engineering manager once told me that “back in the day, SREs got promoted for writing monitoring systems, and so we ended up with a bunch of unsupported monitoring systems”


Not really, it's barely used anymore. Monarch has replaced it for most situations.


I came across the Pants build system [1] the other day, which looks like it shares some similarities. Though currently very specific to Python.

The value proposition is also a bit more clearly defined there [2]

[1]: https://www.pantsbuild.org/docs/welcome-to-pants

[2]: https://www.pantsbuild.org/docs/how-does-pants-work


Hey, Please maintainer here! One of the big differences to the other Blaze like build systems is Please actually doesn't treat any of the built in languages any differently to other languages. They are implemented entirely in the build language which is totally open to you as a consumer of Please.

Another big difference is Please is written in Go so there's no dependency on VMs or runtimes.


Bazel is moving in that direction too—the C/C++ rules are built-in, but they’re slowly being extracted into Starlark.


Why are landing pages so incredibly uninformative?

In practice, I spend under two minutes to click around and see if can 'get' it. What does it do especially well? What's the syntax look like? If you are brave, what doesn't work well?

No idea why this exists or why I would use it. Bad website. No cookie.


The number of times I've said "Please build" to myself, over the decades ...


Admitting ignorance here, I have never intentionally used a build system like this. My normal process for e.g. Python deployment is to write a Dockerfile that installs the dependencies, installs packages, and then copies my user code in. Then I use Pulumi to upload that image to a Docker repository & deploy it to a k8s instance.

What am I missing by doing that? This looks really slick, but I'm not sure how, why, or where to use it.


Please excels in a mono-repo situation. We built Please because we struggled to get Buck to handle Python, Javasript, and Java while having protobuf code generation for all these languages.

Fast forward 5 years, now we're using Please to build all our code, generate hashes for docker images, template those hashes into our k8s .yamls, generate the documentation website from the docstrings in the proto files etc. etc.

With a language specific build system, you would have a lot of trouble handling things like this.


Speaking from my own limited experience, plz works really well for compiled languages, and total-build approaches (everything from source). I am not sure how much of a multiplier it is for python, apart from finer-grained dependency control and good sandboxing


Those kinds of systems are for building stuff, they assume and have multiple build steps with complex dependency relations (like autogen header -> Compile -> link -> test), and that each step takes dozens of seconds and maybe even minutes.

In your case, it seems you only have one non-trivial step: python dep install. So this system will be quite overkill for you.


> If you're familiar with Blaze / Bazel, Buck or Pants you will probably find Please very familiar

Yes, so why would I use Please over any of them? I've spent close to 10min reading and have no idea why this exists or why anyone would use it. It looks like Bazel with a different config format, in which case why wouldn't one just use Bazel?


From their FAQ: >Why use Please instead of Bazel, Buck or Pants?

>All four of these systems are quite closely related in the scheme of things, being inspired by (or in Bazel's case, a direct open sourcing of) Google's Blaze.

>Several of us had worked at Google and used Blaze extensively there; we were excited about it being open sourced as Bazel but by then we were already using Please internally. It's a great system but we have slightly different goals, specifically we're aiming Please at being lighter weight and pushing the boundaries of what can be done within the BUILD language. Since Please is written in Go there's no runtime dependency on the JVM.

>We actually used Buck for some months internally before deciding to write Please and before it was capable of building our repo. We preferred it to other options available, but again we're focused on different goals; Please is easier to extend to new languages, has a bunch of features that we specifically wanted (e.g. test sandboxing) and has a stronger focus on BUILD language correctness. Conversely we have much less support for mobile development.

>We're least familiar with Pants; one of our developers briefly tried it and while we liked many aspects we didn't think it was the ideal fit for us at the time.


That's a lot of text to say three things: doesn't use JVM, has test sandboxing, and the DSL is "more correct". This doesn't tell me what I want to know.


I've only used Bazel in anger (in a medium-small sized project); I've found it really frustrating because it effectively has two separate languages (inside build files and in custom rules), even though it's both Starlark. It's especially terrible for doing things that Bazel didn't already know about (i.e. running shell commands to generate things, even if you could tell Bazel all the inputs and outputs).

We wanted to switch away from Bazel because of the mismatch; it looked like Pants and Buck had the same issues. At the time Please didn't have sensible documentation so it was skipped. From what they have now it seemed like it's more reasonable in this respect, but I don't actually know yet; perhaps somebody who knows can chime in?


What do you mean? Genrules are very good at running she'll commands (or even entire executables).

By custom rules do you mean macros, or full on custom rules?


Genrules are okay for commands, but most things I want to do end up being tiny shell scripts, which isn't a great fit.

I mean full on custom rules, exactly because of the inability for genrule to span more than one command (so we end up having to generate tiny shell scripts and running them).

I've also had issues debugging things any time I've tried to use genrule; the error messages tend to be unhelpful in my experience (though that's been a while and I don't recall the details). Things that look like they'd do what I want would just not work in unhelpful ways, and interrogating the system usually didn't work until there were no errors (at which point I wouldn't need to interrogate it anymore…)


Genrules can run shell scripts though, like most of the time my genrules invoke an arbitrary python script. Shell would work much the same.

What do you mean by a "command"? I'm very confused by the nomenclature here.


I hate bazel with passion, I have to use it at work on really large project, and the UX is so wrong... built for google by google and my CTO thinks we can be google if we use it...


What specifically about the UX is annoying to you? I'd be interested in hearing what could be done to improve the experience. Is it just writing BUILD files or is it the cli interface?


Some people really dislike the JVM, and Please is written in Go.


Exactly what I was about to say. Bazel is a monstrosity in terms of how much shite it pulls in to get started.


Is it a monstrosity? You need the JVM but the JVM doesn’t seem like a monster to me.


Bazel doesn’t need the JVM as it’s built into its self contained binary.


Hm, I guess my point was that it still needs the JVM, it just uses the bundled version. I guess that wasn’t clear.


Yes, but the bundled JRE is slimmed down, to the point where, last I checked, the bazel standalone binary is ~50Mb.


50mb.

"slimmed down"

lol.


You might not want to look at some golang binaries ;)


The JRE is 655.8M on my system.


And I would have considered that to be large 20 years ago, because it would have filled an entire CD-ROM disc, and it would have taken over five hours to download.

However, I still have somewhere over 100 GB free on my seven-year-old laptop, and the download will take me ~40 seconds.

But even if the 600MB were a problem, Bazel doesn’t need an external version of the JVM. It bundles its own, and fits the whole thing in under 50MB. Some people are under the mistaken impression that you have to install the JVM first and then install Bazel, which is simply not true.


Good to know that you live in a privileged place and access a fast internet connection. Most of the places don't have that kind of bandwidth [1] to download 600 MB of file in 40 seconds. Also, Bazel needs a huge amount of RAM because of JVM and runs a daemon process in the background to speed up the build duration. Running a background JVM daemon process is a NO for me and Bazel wastes system resources.

[1] https://en.wikipedia.org/wiki/List_of_countries_by_Internet_...


> Most of the place don't have that kind bandwidth…

Most people don’t have the kind of bandwidth it takes to download a 50 MB binary? Are we still talking about Bazel here, because that’s the size of the download. What about tools like compilers? You have to download those, too.

> Running a background JVM daemon process is a NO for me and Bazel wastes system resources.

What system resources does it actually use up? A half-gig of RAM? This kind of optimization is penny-wise and pound-foolish. You are spending your precious time and energy worrying about a resource whose marginal cost to you is about the same as a cup of coffee.

I run Bazel on a terribly obsolete, seven-year-old laptop which is my daily driver. Sometimes Chrome will choke on a website, sometimes I'm waiting ages for Homebrew to update or for NPM to download some packages, but Bazel is not a problem.


> You are spending your precious time and energy worrying about a resource whose marginal cost to you is about the same as a cup of coffee.

I don't install tons of random garbage because I want to know and understand what is running on the hosts that I maintain.

I need to be able to debug the stuff I run, and complexity makes it difficult.

Tenths of MB of stuff is complexity. Tenths of build time or run time dependencies is complexity. A compiled tool when a script would have been sufficient is also complexity.


> I need to be able to debug the stuff I run, and complexity makes it difficult.

Bazel is way easier to debug than Make. Debugging a decent-sized build system made with Make is just an exercise in suffering.

I am… honestly… no longer interested in understanding the entire software stack. I understand my time on this earth is limited and want to spend it doing other things. With Bazel, I am spending less time fucking around with build systems and more time doing the stuff I care about.

> Tenths of MB of stuff is complexity. Tenths of build time or run time dependencies is complexity. A compiled tool when a script would have been sufficient is also complexity.

As someone who still writes C, I can understand the joy that people feel when you make some cool program and it’s measured in KB, not MB. However, what you’re describing strikes me as fetishistic.


That 600 MB is mostly standard library, and reusing it makes every app smaller. Statically linking everything creates size and version skew problems that I hoped we had left behind in the 1980s.


So you can't spend a one time cost on the scale of downloading Vim 10 times?


50mb for a build tool is still ludicrous. Very few product codebases will ever reach that size.

In my experience there is a strong correlation between executable size and how hard it is to unravel any issue that arises in it.


> In my experience there is a strong correlation between executable size and how hard it is to unravel any issue that arises in it.

Bazel bundles the JRE, that’s why it’s 45 MB.

The correlation is kinda beside the point, though, because my experience with Bazel is that it’s easier to unravel issues with Bazel than unravel issues with Make, and Make is much smaller.


Bazel has its JRE built in its self contained binary. Not sure what you’re referring to.


I also was trying to figure out why I would use this over Bazel. Then I remember reading a story yesterday about how ex-googlers miss their tooling when they leave. Their CTO is an ex-googler, so maybe this is the reason.


But Bazel is developed by Google itself right now. Presumably if you missed Google tooling you'd rather use Bazel, no?

I'm not suggesting Bazel is perfect: I think e.g. Starlark's insistence on being a separate language does more harm than good. (FWIW I also think the JVM objection is a little silly.) But I am saying preferring the Google tooling would ostensibly mean you like Bazel a lot already :)


I've read Bazel without the rest of Google tooling (giant monorepo of everything, distributed build farms, etc) the experience is nowhere near the same.


What do you mean with "insistence on being a separate language"? Many build tools have their own DSL.


Starlark is _almost_, but not quite_ Python, and Python 2 at that. You could do everything Starlark does in actual Python and get stuff like static type checking for free.

I think the term DSL is overloaded here? Consider all the Lisp and Ruby stuff that's definitely DSL but doesn't need most of a language implementation.


In a word: simplicity

Bazel has a lot of magic and crazy abstractions. Please is far easier to get your head around and as a result, far easier to bend to your iron will.


Please predates Bazel IIRC. It just didn’t gain as much traction. Please seems considerably easier to operate, but it doesn’t enjoy Bazel’s rigorous usage (although I’ve only encountered bugs when trying to use Bazel to build Python 3 projects).


Bazel’s Python integration quality is a level down from its Java, Go, and C++ integrations. It’s getting better though.


The landing page on desktop is impossible to read, random text spread all over the screen, no idea what's the intended text flow.


At some point it becomes actively harmful to build and publicize tools that further the fragmentation of an ecosystem. We need a unified build system, not yet another build system. To get there we have to contribute to existing tools to keep making them better.



If I was building a company today I would use Earthly instead of Buck, Bazel or Please. I think Earthly gets out of the way of actually low-level builds and instead focusing on high-level, Dockerised targets. The benefits range from allowing proper IDE support, to very easy system/integration tests, while still reaping the benefit of remote execution and aggressive caching.


Wish there was a new generation high level build system like this or Bazel or something but with decent JavaScript/node_modules support.


The Bazel and JS story is in its infancy. Things will get better once certain big companies have time to adopt and mature into it.


This looks great. A minimalist no-BS version of Blaze that supports everything I would need is something I'd be happy to use.


Hey guys! Thanks for checking Please out! I see a lot of recurring themes in the comments so I thought I would clear some things up.

What is Please?

Please is a multi-language build system designed for huge mono-repos. It was created by a couple frustrated ex-googlers who were familiar with Blaze (which was later open sourced as Bazel). We found the "real world" alternatives to be somewhat lacking and so Please was born!

Please draws inspiration on the Blaze paradigm. If you're familiar with Bazel, the biggest difference is Please aims to be simpler and have far less magic in the binary. We push the implementation of the build rules into the build language, dog feeding them to ensure Please is flexible enough for any task. Also Please is written in Go so doesn't require a JVM ;)

If you're not familiar with Blaze/Bazel, here's what all the fuss is about:

1) Hermetic builds: builds are run in their own tightly controlled environment. Each step of the build runs in their own temp directory isolating them from other steps and only having access to the files and environment variables they've declared as their inputs. Please also has sandboxing built in taking advantage of the linux kernel to further isolate tests.

2) Scalability through incrementallity: if you've used Make, you're probably familiar with caching problems. Make uses last modified timestamps on files to determine if they need to rebuild each step, which turns out is fallible. Please uses a hash based approach which is far more robust. Most of our developers don't even know how to clean the cache. As a result, we can incrementality build our entire repository locally and on our CI workers no matter how big our repo gets.

3) Flexibility: the build language is a dialect of python. This can be used to write "build definitions" which define a unit of work i.e. compiling a Go package. There's nothing special about the built in definitions; it's totally possible to write your own to automate nearly any part of your development process. You could generate code, template kubernetes .yamls and beyond!

4) Unified developer experience: The please command line provides a unified experience across your codebase. Want to test all the tests under a branch of your repo? `plz test //some/part/of/the/repo/...`. It doesn't matter what language you're using, what those tests depend on etc. etc. Please can always run them for you.

PS: Apologies for the website. We're a small team of build system engineers, not front end types. If you want to offer your skills, I'd be happy to point you in the right direction: https://github.com/thought-machine/please.


Can somebody sell me this tool while comparing it to `make`, or Elixir's `mix`, or Rust's `cargo`?


If you are working in an environment that needs to mix between different languages, and a monorepo makes sense for you, then a tool like please let’s you stitch those things together and handles dependencies between those things. So if you have a service defined in an IDL like gRPC or OpenAPI, with a backend in python and a web front-end in JS, and API docs, etc, a tool like this can stitch together changes really efficiently.

Please is a lot more lightweight than Bazel, so it’s easier to get deployed and work with different projects. The new maintainer has been shaving off a lot of warts lately and it’s getting a lot better.

If the above doesn’t describe your situation, then a tool like please, pants, buck, or bazel is probably not for you. And that’s ok too.


Or CMake or the C/C++-oriented http://floooh.github.io/fips/ (a high-level build system "similar to Rust’s Cargo or Javascript’s NPM, but for C/C++ projects.").


Have used it in a go mono-repo. It’s own convention on packages and bad integration with IDE makes it a hard sell for me. I mean I just paid for Goland IDE why would I waste my time integrating with something that might or might not work for others. It’s a very hard sell.


Go tools in general have poor integration with build systems other than "go build". One trick I find useful is to use Go modules (and therefore goimports, not goreturns) and structure your build so that it looks like a standard Go module build, even though it's really being built by some other system. In other words, it's a mix of making the build system conform to Go norms and making your IDE adapt to the remaining differences.


I recently added a config option so plz test will run tests in the package directory rather than the repo root (just like go test). Should make things a little easier.


Would you consider it for other mono-repo language projects?


How big does the project have to be in order for this system to have benefits over Make?


The problem with Make is that it doesn't know what it's building, so it can't do anything smart. For incremental builds to work at all, you have to supply the smarts. Having worked on a number of make-based projects (I am most scarred by buildroot), I can tell you that people make mistakes with build rules. The project then devolves to doing a clean build for every change, turning what could be a few milliseconds of CPU time into a 30 minute rebuild.

The idea of these build systems like Bazel is that the rules are correct so that you don't have to worry about writing correct rules, and you have a high probability of an incremental build producing a binary that's bit-for-bit identical to one from a full build. The result is that you don't do full builds anymore, and save the latency of waiting for things to build. (That latency shows up in the edit-build-test cycle, how long it takes to deploy software to production to fix an emergency bug, etc. So it's important!)


I am personally not convinced that any build system can be correct and general, but perhaps that’s my lack of experience speaking.

On that 30 minute note though: so, how big does the project need to be in order for Make not to be enough? And at that size, why wouldn’t the project invest the extra week it takes to get the Makefile correct?


It's easier to adopt a correct build system when your codebase is still simple. It can save you time, because it can prevent lots of unintentional errors over time (e.g. you might not notice when you get an incorrect result after an incremental build).

In my opinion, this is a bit similar to languages with static vs dynamic typing.

If your current setup works well for you, it makes sense to keep it, though.


I completely agree with you. To give the build system the understanding it needs, you have to give up the generality.

For me, I've found that the results are best when you use a build system and you use it exactly the way the author intends. For example, Go's built-in build system is so good that you don't even notice it's there. It automatically updates its configuration as you write code. It does a perfect incremental build every time, including tests. The developer experience is basically perfect, because it has a complete understanding of the mapping between source files, dependencies, and outputs. But, it is not extendable, so you're screwed when your Go project also needs to webpack some frontend code, or you need to generate protocol buffers (which involves a C++ binary that runs a Go binary), etc. So, people bolt those features on, and the build system becomes more general, but not quite as good. (Then there's make, which is as good or as bad as you want it to be.)

I think super small projects often do get their Makefiles right. But you can manually build small projects with something like "gcc foo.c bar.c -o foobar -lbaz", and so you don't really benefit from any improvement over the bare essentials. (Nothing wrong with keeping your projects tiny in scope, of course!)

But, sometimes you don't have the luxury of a super small project, and the Makefiles become quickly unfixable. Like I said, I am most scarred by a buildroot project I worked on (that's embedded Linux, basically). It never built what I expected it to build, and to test anything reliably I either had to yolo my own incremental build or wait a while for a full build. My productivity was minimal. I could switch between client-side and server-side tasks on that project, and so I really only touched the client if it was absolutely necessary. I would never be productive enough to undertake a major project that truly added value with that kind of build system, so I let others that didn't have the server-side experience write the client-side stuff. In that case, the poor build system silently cost our team productivity in terms of artificially splitting the team between people who could tolerate a shitty developer experience and those who couldn't.

I don't think anyone has fixed the buildroot problem, either. If you want to build a Linux image today, you are stuck with these problems. Nothing else is general enough to build Linux and the associated random C binaries that you're going to want to run.


Yeah, that tradeoff between generality and correctness really seems tyrannical.

It kind of feels like the best trajectory is to start small with a general build system, and upgrade as needed? And then if you are confident the project will grow, starting with the specific build system fine too.



Does Please allow one to easily specify a specific JDK version to use to build and run tests?

I wasted a lot of time getting Bazel to use a modern JDK.


Would be interested how feasible it would be to replace Jenkins with this?


Please is a pretty different tool to Jenkins - you could use it within Jenkins, but instead of make / Gradle / go build / cargo / etc.

It does have some features to facilitate CI and testing at scale, e.g. `plz query changes` can be used to find a minimal set of tests to run for a PR.


Is there any help from Please on building TypeScript projects faster?


It's pretty easy to wrap the typescript compiler however I've not seem many people use pure TS like that. Mostly people want to use webpack and TS together. Unfortunately webpack doesn't expose anything that would allow us to incrementally build a webpack bundle.


> Indentation is normally four spaces. Tabs will be rejected by the parser.

You love to see it.


Title can possibly be modified to mention that this is a tool to build code.


For what it's worth, I think the too-short title is pretty normal for HN. I'd blame its less than optimum informativeness on HN's cultural norm of short link titles, as reinforced by the administrators' moderation.


> Please (please.build)

'build' kind of indicates that


Unfortunately for those of us following HN through RSS readers (like Feedly) the URL doesn't show until you mouse over the link ... but even then, I agree adding that additional piece of information such as "build software" would be useful.


Feedly has an API, I started writing a script a while ago to improve some RSS feeds, including HN (by making the number of comments and the URL appear in the body), but never finished it.

If someone had the same idea, please don’t hesitate sharing.


Most RSS clients don’t have that bug.


Since far from all TLDs have reliable semantics and it's impossible to know all of them, it doesn't. And compiling a project from source code is not the only meaning of "build", not even in tech.


> > Please (please.build)

> 'build' kind of indicates that

Not the GP, but as a developer reasonably conversant with build systems and their pain points, I still assumed this was going to be some form of lazyweb[0] related blog post or portal rather than a build system named 'Please'.

[0] https://ftrain.com/dear-lazyweb


With the URL, I was expecting an opinion piece along the lines of https://a16z.com/2020/04/18/its-time-to-build/.


From the URL, I thought it was going to be a joke page about developers praying "come on, please build!" at complex build systems.


[flagged]


I don’t think this is a fair reason to write the tool off. They support Homebrew, and I’m sure will add other install methods in the future. Piping to bash is no worse than clicking “yes” on every step of an install wizard.


The alternative is not “install wizard”, it is either “here is a deb file with no post* scripts” or “here is a tar.gz, extract it anywhere and add symlink”

Both of those are vastly safer, because they do not require root privileges, and guarantee to uninstall cleanly.


This. Simply the decision to eschew standard package management already says unpleasant things about their approach to integrating with my environment, or in whatever it is the installation does being reversible using standard tools. "Install wizards" do indeed have exactly the same issue, which is why those are also terrible.


Often these "pipe to Sh scripts" support --help and a variety of configuration options anyway. The benefit of a script over a binary installer at least is that you can inspect the script before running it!


Have you inspected one of these scripts? What have you found? (I've tried it a few times and haven't felt like I learned anything meaningful from doing so.)


I almost always inspect these kinds of scripts before running, more out of curiosity than anything, but also so that I know its not going to do that so stupid that even I can see it's stupid. Usually you can just pipe to `cat`, which is super low effort to do.

I've occasionally seen scripts that install some other application where it was not clear that it was a dependency and there was no heads up to the user that this was going to happen: that kind of behavior makes me more distrustful of the author, so there's a useful signal there.

Most scripts like this seem to amount to "configuration defaults + cli completions installation". To that end, I find looking at them useful because it gives me a sense of the expectations of the tooling and hints at where I might find things if I need to go debugging later.

When they are provided by the same entity as the program you wish to run, I don't see how it's significantly more risky to run these scripts than running the application code itself with the active user's permissions. Still, if there was something, by glancing at the script, you have half a chance of seeing it. If there's something there that doesn't make sense, that's both a "proceed with caution" and a learning opportunity.


cat to terminal is kinda not enough. I can own you very easily if you do just that and think you've seen all the code that will be executed.

Save to file, and read the file in the editor.


It’s not really intended as a defense against being owned per se, it’s more about knowing what’s going on and getting an additional signal about the risk profile (not just from maliciousness) of the thing I’m about to run.

That said, I generally pipe to file and cat the file, yes, if only because it somehow feels wrong to download it twice.


I understand, but cating the saved file or printing from pipe to the terminal directly has identical issues. Terminal ANSI escape sequences are interpreted either way.


Great point. Thank you.


I did!

The usual annoying thing is the automated package install. I have not looked at this particular package, but in the past, I have seen:

- installing specific gcc version and making it system-default.

- installing “virtualbox” packages - this was the worst, as the machine had a few KVM VMs running at the same time, and KVM and VirtualBox cannot run at the same time.

In general, I now stay away from every install script, and if I cannot, then I run it in docker container. Life is short, I do not want to spend it fixing my primary workstation.

(And I examine postinst scripts in deb files too, but those are usually much more boring)


Yuck, those are some really badly-behaved installers.


Yes, plenty of times. Usually I find that they do what you would expect them to do: set up a bunch of parameters like an installation prefix, and then copy files around. They also handle user input options and maybe prompt for some stuff.


This kind of dismissive, even arrogant, comment has no place in this community.


Do you think clicking "yes" on an installer is any more secure?


Yes. For starters, it's much harder to suffer disastrous consequences from running truncated command due to terminated network connection when running an installer.


This is easily rectified by script authors by adding a main function invoked at the end. At a certain point, the argument simply becomes "executing arbitrary code is potentially dangerous", which is trivially true, and also not a helpful security posture for anyone who wants to use their computer pragmatically.


Theoretically yes, it is all turing complete at the nd. Practically no, it is the matter of established conventions.

For example, most interactive installers I have seen prompt for install location, and write minimal amount of files outside of it. The chance that they break your system is pretty low.

At the same time, many |bash scrips like to install system-wide packages, or use pip as root, or other potentially breaking things. And they do not ask you before installing this, and often auto-confirm, too.


Why? I pragmatically use my computer with packages my distro offers me. When I download something from untrusted sources, I do take a quick look at what it is before executing it. It still is dangerous since I'm not able to spend time on verifying every single line of it, but it's up to me to decide whether that's an acceptable risk or not. All perfectly pragmatic.

I'm not a kind of person who's on a crusade against curling into bash, but I never do that myself because it feels dirty. It gives me no advantages and makes me unable to even take a quick look to reassure myself that things at least look like what I'd expect them to be.


I also read scripts from the user repositories on my distro. It’s perfectly pragmatic for us because we understand shell scripts and system administration. It makes me feel better too. But I would never tell someone to avoid installing developer tooling simply because it pipes curl to bash. It’s not pragmatic to tell a new developer they need to learn to be a sysadmin before installing homebrew.


That's a strawman though. Nobody wants to tell that to new developers. This is all about conventions and normalizing bad behaviors. New developers should avoid piping curl to bash and explicitly download files instead because when they eventually will become experienced enough to inspect those files they will start doing it naturally.

Good practices are rarely black and white, they're often there to influence how you think about stuff and what habits you develop.


I personally despise people combining terms like "fast" or "lightweight" with languages like Go and Python. We are looking at a build system that, to my knowledge, doesn't take any advantage from being written in Go, other than "being easier to write and understand by a human".

When I see someone giving such reasons for not using a more performant language, I immediately associate such project with janky, fast written codebase and disregard for proper, complete documentation. "Because you can just read and understand the code", yeah I can do the same in well maintained code of a faster language.


> doesn't take any advantage from being written in Go, other than "being easier to write and understand by a human"

I would assume that is has the same advantage as most other Go programs: It's easily distributed as a single static binary, and doesn't need a heavy runtime to be installed.


No thanks, I want my binaries to use shared objects so that an entire OS can receive quick security updates.

And I want the binaries properly packaged with metadata, documentation, manpages to keep the host secure and tidy.


IMHO the ideal situation is that both distribution mechanisms are available; single file static binary and also packages for different OS's.

Which one you want to use in a given situation is contextual.

Murphy's law seems to dictate that the version of the build tool lovingly cared for by the OS's package manager is never the version you actually need.

Build tools in particular have a slightly different lifecycle to many things running on a host, which are "owned", cared for and configured by a system administrator.

Usually the project you're building "owns" the version of the build tool it needs and there's no guarantee that it will build with a different version. As a consequence, genuine build tool security issues must first be handled by the team making the tool and then by the projects depending on it when they verify compatibility.


> I want my binaries to use shared objects so that an entire OS can receive quick security updates.

Have shared libraries historically improved security much? Debian helpfully "updated" OpenSSL, and thanks to the magic of shared libraries, every program on a Debian system generated weak keys.

Personally, I don't think there's enough software on the average filesystem image to justify sharing libraries. Most stuff "in production" these days is in a container, which shares nothing with the other containers running on the same VM, which share nothing with the other VMs running on the same bare metal server. In the case of containers, a full rebuild from source is just as easy as updating a shared library -- maybe even easier. So the "quick security updates" doesn't seem to justify the high maintainability cost of shared libraries. (The high cost is having to maintain binary compatibility with applications you've never even seen before.)

The whole reason things like "Linux distributions" exist is for some central authority to coordinate and handle all the breakages this model causes. It's nice that they do it for us, but it's kind of makework to save a few bytes of network transfer now and again. Doesn't seem worth it to me.

> I want the binaries properly packaged with metadata, documentation, manpages to keep the host secure and tidy.

You've just invented containers. Many other people agree with you, which is why they became so popular.


> Debian helpfully "updated" OpenSSL, and thanks to the magic of shared libraries, every program on a Debian system generated weak keys.

I'm going to guess you never checked how many vulnerabilities have been fixed in Linux distributions before a release, or backported to existing releases: hundreds of thousands.

Cherry-picking a single incident (involving a library with a history of vulnerabilities) is hardly meaningful.

> Most stuff "in production" these days is in a container

It's provably not, outside of the SV bubble. The large majority of software in the world is still deployed traditionally.

> So the "quick security updates" doesn't seem to justify the high maintainability cost of shared libraries

A good number of large companies think otherwise. Also, people working in security.

>> I want the binaries properly packaged with metadata, documentation, manpages to keep the host secure and tidy. > You've just invented containers.

No, these are OS packages and predate containers by many decades. This is getting silly.


I too used to take this view until I worked on a system which was on a limited storage budget of roughly 500MiB. Shared libraries go a long ways on a system like that towards making the entire thing fit. The other option was linking multiple binaries together busybox style, which also works wonders, although it forces release coupling so there are tradeoffs. Not every linux system is for desktop or server either.


> doesn't need a heavy runtime to be installed

Of course it does. The runtime environment just happens to be installed alongside every Go executable by virtue of being statically linked to them.


I think hobofan means that the Go runtime is lightweight when it comes to runtimes.


Are build systems typically CPU bound, even during heavy operations such as calculating dependency closure (or other things I'm unaware of)? If not, the I don't see a need to use a language that doesn't provide high level, somewhat costly, but easy to use and understand abstractions. And if it's faster than its competitors and it weighs less, then there's also nothing wrong with calling it "fast" and "lightweight" (though I have no idea whether please fulfills this)


Build system performance tends to be IO-bound, but responsiveness for things like no-op builds tends to be CPU-bound -- and the JVM in particular really suffers. Optimizing IO performance makes it perform better, while optimizing CPU performance makes it feel better to use.

Arguably, making tools feel better to use is more important than raw performance. When you're doing a no-op build, the difference between 100 milliseconds and 500 is large, and the difference between 10ms and 100 is enormous.


> Optimizing IO performance makes it perform better, while optimizing CPU performance makes it feel better to use.

Optimizing I/O performance certainly makes building software feel better though. Running make on a tmpfs directory feels absolutely amazing. It feels even better on subsequent runs since Linux uses free memory to cache files.


My full time job was to make Bazel more efficient for about 1.5 years. The answer is: it really depends.

If you are executing 100,000 novel parallel C++ actions, then compilation for that might be I/O bound? If these actions are all cached though you would likely be CPU bound instead, as the build system works hard to discover that all the work is already done.


Please don't do programming language flamewars on HN. They're tedious and we're trying to avoid that here.

https://news.ycombinator.com/newsguidelines.html

We detached this subthread from https://news.ycombinator.com/item?id=25240535.


Go is pretty fast, within factor of 2 compared to C/C++.

Python is another thing. Most CLI utilities written in python are unusably slow.


[flagged]


It’s not just a choice. It’s also the kind of tooling the people developing the system have at their disposal.

For example, if all I have is a MacBook running macOS and Docker, how can I reasonably expect to port this to Windows? I haven’t owned a Windows machine for 10 years.

I am the author of Buildbarn, a build cluster implementation for Bazel/Pants/.... Buildbarn also doesn’t support Windows. If someone sent me high quality patches for Windows, I’d merge them for sure, but nobody has. Sorry you are ‘stupidly disappointed’ in me.


What are you doing that requires kernel specific code? Shouldn't you just be using cross platform libraries?


I make use of openat() and friends to safely populate and interact with build environments on disk.

I perform scanning of process tables to kill lingering processes. Go has no functions for that in its standard library. There are some third party libraries for that, but they are all of pretty poor quality. They are slow.


It's open source, if you want it, you can add support for it. Which is a lot more than what you can do for a lot of other development tools that only run on Windows and have lackluster or no support on Linux.

Also, they might just not have Windows developers or a use case for running the tool on Windows, so even if the cost of adding Windows support was zero, they'd have to go out of their way to test it. A lot of open source code might run just fine on Cygwin, but most people choose not to support Cygwin because damn near no-one uses it anyway.


Devs on Unix are by the far the worst when it comes to writing cross platform code.

This is written in go! How isn't it cross platform?!


Maybe if windows were free to use, folks who are volunteering their time and energy would be more prone to support it. Alas...


Macos is free to use?


Is this type of response called for?

Please be a good netizen.

If you’re interested in the project and are a Windows user, contribute to the project.


I can see your point, pretty aggravating. Is WSL an option?


Hi, one of the implementors here... Yes, WSL is an option, and I believe Please works just fine in it. Right now we don't have any CI etc set up so we can't stand behind it and say "this works", but it works in WSL as most Linux things do.


Great stuff, thanks for this great addition to the build ecosystem




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: