Defining a Skylark py_proto_library via proto_library

[htuch]

Given #2626, I was wondering whether it would be possible to add a pure Skylark py_proto_library built on proto_library.

I've played around with rules and had a look at how rules_go does this with toolchains (https://github.com/bazelbuild/rules_go/pull/803/files). These basically involve you maintaining an additional dependency tree (which proto_library is supposed to solve) and having your custom py_proto_library applied across this tree. The most promising approach to avoiding this seems to be to use aspects.

It's pretty clear how aspects could be used to invoke protoc to generate the _pb2.py files. What's less clear is how to use these outputs, which are unknown ahead of time and come from the proto_library transitive dep provider, in conjunction with the native py_library rule (I think this is the sandwich problem?).

Any guidance here from the Bazel proto or Python teams?

[dslomov]

The idea here is that:

• aspects will collect all the transitive _pb2.py files and pass it to py_proto_library rule
• py_proto_library rule will make those available to py_library rule in the py provider.
  This is not super-wel documented.

[htuch]

@dslomov can Bazel team do a worked example of using aspects + providers to implement a py_proto_library as a blog post? When I think back to the most useful docs I've seen on Bazel, I think these extended examples are great, e.g. I learned a lot about repository_rule from following the C++ autoconfigured toolchain example linked to from https://docs.bazel.build/versions/master/skylark/repository_rules.html.

[ali5h]

any update on this would be really appreciated

[junghoahnsc]

Which rule should I use for now?
Should I use https://github.com/google/protobuf/blob/master/protobuf.bzl#L316?

[gnossen]

@dslomov @brandjon
Ping. It seems this feature is in limbo between the gRPC, Protobuf, and Bazel teams. Any updates on the Bazel end? A conversation revolving around it is unfolding at grpc/grpc#19183.

[gnossen]

@dslomov @brandjon Friendly ping.

[brandjon]

+@hlopko since I think there's plans about prioritizing proto support in general.

[hlopko]

Yes, I'm trying to get us a person to work on proto rules. No concrete results yet, only active effort :)

[sseveran]

Any updates on this?

[hlopko]

Yes, we didn't get the person :/ I'm actively trying to prioritize this, but more important stuff keeps coming up.

[nicknezis]

Why not leverage the work being done here? It seems they are handling Python and a few more languages. https://github.com/stackb/rules_proto

[pcj]

Also worth noting that grpc folks did implement py_proto_library and py_grpc_library since earlier comments. I've used them and they work on a routeguide example but have not used them extensively.

https://github.com/grpc/grpc/blob/master/bazel/python_rules.bzl#L88

Here's link to the functioning example: https://github.com/stackb/rules_proto/tree/5050a8a13d6d13d4284c0cc9f3711181f532f4bb/example/py_grpc_library

[awbraunstein]

What's the Bazel team's philosophy on rules like the py_proto_library rule being requested here. Currently there are implementations from the protobuf team, grpc team, as well as other users such as http:https://github.com/stackb/rules_proto mentioned above. Is the end goal for the bazel team to own these rules? Or should they be owned by some other entity?

As a user of these rules it becomes difficult to understand which one is the best/right one to use since they all tend to function in the same way (except some use the proto_libaray and some don't.)

[dayfine]

As far as I've tried, none of these py_proto_library works to my satisfaction (so are some other rules that I've tried, like go_proto_library). My expectation for these rules come from google3, and it's frustrating the bazel versions don't quite work the same way.

What I would expect with a lang_proto_library and which is not currently how these rules work:

proto_library(
    name = "foo_proto",
    srcs = ["foo.proto"],
    deps = [
        "bar_proto",
        "@com_google_protobuf//:timestamp_proto",
    ],
)

py_proto_library(
    name = "foo_py_pb2",
    deps = [":foo_proto"],
)

proto_library(
    name = "bar_proto",
    srcs = ["bar.proto"],
)

py_proto_library(
    name = "bar_py_pb2",
    deps = [":bar_proto"],
)

# Using foo_py_pb2 without fuss
py_library(
    deps = [":foo_py_pb2"],
)

Basically:

1. lang_proto_library only needs foo_proto in dependencies, not other bar_py_pb2. The Go rule and some flavors of the python rule do not work like this.
2. When using the proto foo in python, I only need foo_py_pb2 as the dependency of the users, i.e. the target would include all its transitive dependencies. The grpc version of py_proto_library fails with this.

[scal444]

Has there been any action here? Agree with the last 2 comments, each version has its own quirks and does not align with internal Google use, which is quite frustrating when trying to maintain open-source libraries that Google also uses.

[weichunstevelee]

Hi I'm curious about the current state. Reading the current implementation of py_proto_library in protocobuffer repo, it says

"""Deprecated alias for use before Bazel 5.3.
    Args:
      *args: the name of the py_proto_library.
      **kwargs: other keyword arguments that are passed to py_library.
    Deprecated:
      This is provided for backwards compatibility only.  Bazel 5.3 will
      introduce support for py_proto_library, which should be used instead.

https://github.com/protocolbuffers/protobuf/blob/main/protobuf.bzl#L636

Did it happen?

[cnsgsz]

I found this doc which has a nice summary of all the options.
The multi-language rule sets seem close to @dayfine said.

Between rules_proto and rules_proto_grpc:

Choosing between the two rulesets will effectively come down to your preference for using Gazelle within your project and the languages supported by each.

Does anyone know if using gazelle is a plus or minus? I assume rules_proto uses gazelle but not rules_proto_grpc. rules_proto_grpc seems to support strictly more languages.

[comius]

Starlark py_proto_library is available in https://github.com/bazelbuild/rules_python/blob/main/python/proto.bzl#L21