Allow overriding of namespace per-language or on the command line

[mmastrac]

In generating code for multiple languages, I've found that a single namespace tends not to work across every single language. I'd love to have a way to either provide per-language namespace declarations or override them on the command-line.

Strawman approaches

Per-language namespace:

namespace[js] fb;
namespace[java] com.mycompany.protocol;
namespace[go] mycompany.protocol;

Command-line override:

flatc --namespace fb -s foo.fbs ...
flatc --namespace com.mycompany.protocol -g foo.fbs ...
flatc --namespace mycompany.protocol -j foo.fbs ...

Command-line namespace prefix (an alternative to overriding the entire namespace):

namespace protocol;

table XYZ { ... }

flatc --namespace-prefix fb -s foo.fbs ...
flatc --namespace-prefix com.mycompany -g foo.fbs ...
flatc --namespace-prefix mycompany -j foo.fbs ...

[ghost]

I agree that this would be a nice feature. I think doing it in the schema would be better than on the command line, since other code will end up relying on those namespaces.

Namespaces are also used to refer to things in schemas, and one parsed schema may end up generating code for multiple languages, so it be easiest if there was a base namespace for all of them, then we can use the attribute system in the parser to define prefixes for languages:

namespace protocol (java: "com.company", go: "company");

If you're able to make a PR for this, go ahead.

[dictav]

Hello,

First, I don't speak English well, I'm sorry.

I tried to update Parser to support the base namespace of NameSpace on following:
https://github.com/dictav/flatbuffers/tree/namespace-additional-info

Because I want to make GoGenerator work well in nested namespaces (#3927). I'm working on following:
https://github.com/dictav/flatbuffers/tree/fix-go-import

I have noticed. In Go, the base namespace must be only the one per a generation. Because the Go's import path, the directory hierarchy and the other repositories are tightly relative. They cannot separate.

I think that it is better to use the namespace command line option
or to add the new token for the base namespace.

[aardappel]

I don't think I understand's Go's namespace restrictions.. @rw, how does this relate to my proposed attributes above?

[rw]

@dictav @aardappel Go's import paths are restrictive. Read here for more: https://golang.org/doc/code.html#ImportPaths

In particular, Go expects you to use the domain name in packages that are not in the standard library. For example, this is the Go import path for flatbuffers: github.com/google/flatbuffers/go.

Maybe the answer is to have a command-line flag for Go that specifies a path prefix. That way, in the schema, you could specify: namespace protocol (java: "com.company", go: "company");, and for Go you would also have a flag that gives you the domain prefix: github.com.

Does that help?

[aardappel]

Why not namespace protocol (java: "com.company", go: "company.com"); ? Why also have a command-line flag?

[rw]

@aardappel What I'm saying is that the host of the source code is expected to be present in the package name. In your example, it would look something like:

namespace protocol (java: "com.company", go: "github.com/company.com");

[aardappel]

Ok, that's all sorts of f-ed up.
But why do we need a command-line flag? To allow companies to share schemas without forking?

[rw]

To allow companies to share schemas without forking?

Yes, that's a good example of why this is needed. In general, if you fork a Go project (so that it lives at a different path) you have to do a big find-and-replace.

[dictav]

@aardappel @rw
I have thought about some proposals again. As I thought, I think that it is better to use the command line flag without changing IDL.
Read here for more:
https://gist.github.com/dictav/e30d87c9c3fbb41142929dc67c2393b9

(Again, I don't speak English well, I'm sorry.)

[rw]

@dictav thanks for writing this up! I think a command-line flag is the best option. @aardappel what do you think, and what would be the best set of flag(s) to use if you agree?

[aardappel]

Sounds like for Go we'll need a command-line flag regardless. For other languages though, it could be an attribute in the schema.. which would be friendlier since it forces every user to agree on the namespace?

Either way, you can start with a command-line option for Go, since presumably they would all be concatenated anyway.

[aardappel]

Fix was merged: #4222

[xiaohaoliang]

My wish is that: namespace protocol (java: "com.company.package", c: "package");

a command-line option for Go is good. But, I think doing it in the schema would be better than on the command line.

Now for compatibility, I may need
--general-namespace Generate the overrided namespace in Java and CSharp.\n"

@aardappel

(My English is poor, I'm sorry.)

[aardappel]

@xiaohaoliang see the discussion above.. apparently in Go command-line is better because the namespace may be different per user of the package.

Though I guess there's no harm in also supporting the schema version, @rw?

[xiaohaoliang]

@aardappel yes, Now command-line is better , I will to do the command-line:
--general-namespace Generate the overrided namespace in Java and CSharp.\n
so, should I push the code to master ?

xiaohaoliang@79c4342

[aardappel]

@xiaohaoliang if for Java and C# you can already specify it in the schema, why do you also need it on the command-line? I thought this was only necessary for Go.

And if we'd also support it for Java and C#, it be good to use the same option/var as the one used for Go.

[xiaohaoliang]

@aardappel the one schema like namespace package_A to generate cpp files and java files .
the wish is : cpp like namespace package_A{...} , but java need like package com.company.package_A; .
The same idl file to generate different paths and namespaces. So need command-line--general-namespace, like Go but different.

Go : command-line:--go-namespace com.company.package_A
path: com/company/package_A/
generate source_file:package package_A

[aardappel]

Yes, but what is wrong with namespace package_A (java: "com.company") ?

And if we also want a command-line argument, I say lets change --go-namespace into just `--namespace and make it work for any language.

[xiaohaoliang]

@aardappel
lets change --go-namespace into just --namespace and make it work for any language.
I think it is good !

namespace package_A (java: "com.company", go: "com.company"); that is perfect!

[mikemccarty-vertex]

I have a working version that adds a --namespace-prefix <prefix> commandline option. Are you interested?

[aardappel]

@mikemccarty-vertex sure, that's a good first step. Any reason not to also add the in-schema attributes?

[aardappel]

And we should change this code #4222 to use a cross-language option.

[mikemccarty-vertex]

Re #4222: I don't think that option is compatible with what I'm doing. That option is an override rather than a prefix. I couldn't see how to incorporate it without changing its meaning.

[mikemccarty-vertex]

Re: in-schema
I was less confident about how to go about doing that. ;)

[aardappel]

Ah yes you're right, Go is special in that way :(

[stefansullivan]

I just wanted to chime in here. I've started using flatbuffers for python, C++, and Java in at least 3 different repos for multiple projects. Now obviously flatbuffers are inherently a serialization/deserialization protocol so their purpose is to be able to cross memory/language/application boundaries.

I really want to be able to use some sort of consistent namespace convention across all those languages, but they all have their own packaging requirements. For example, java requires com.company.packname.nestedpackage naming convention, while C++ doesn't tend to use namespace company::package. Further, pythons package names are identical to folders, which has been an issue with Mac OS's case-insensitive filesystem more than once 😭. There's also commonly-accepted coding styles that differ between these languages (e.g. CapitalCamelCase in C++ vs under_scores in python)

It would be really nice to be able to generate a schema that used one namespace, but had some convention/language feature to specify how those namespace get translated into directories, packages, etc. for each language

namespace Company.Package.Subpackage (java: com.company, python: company);

Or maybe there could even be a notion of packaging that is required for certain languages? e.g.

namespace package.subpackage (cpp: Company);
package java:com.company, python:company, go:github.com/company;

I don't know. Just now I decided not to use the raw generated sources as part of my build process because I need to rewrite the package names per-language

[aardappel]

@stefansullivan I agree this is a pain. I'm not sure if there would be a builtin way that would make everyone happy, but for now having prefixing options at least let people work around namespace differences.

As for code styles, we have tried to do automatic conversion of identifier styles (e.g. translation to camelCase in Java), but I am not sure how consistent we are in this.

[mikemccarty-vertex]

@aardappel @stefansullivan I implemented in a fork a --namespace-prefix commandline option that would prepend a user-specified namespace prefix (e.g. "com." for Java). I got it to work but I stopped using it in production when I realized it broke gRPC compatibility between different languages. I haven't had time to investigate whether there are reasonable workarounds for this problem. In the meantime, the fork still exists but my motivation to see it merged to master has waned...

[zats]

@mikemccarty-vertex hey I was working on cpp override for the namespaces when I found this issue.

Intuitively I understand that it's better to implement a language independent command line argument allowing to override namespace per specified language but I definitely don't have enough domain specific knowledge to cover all the languages supported outside of cpp

What's the current status of this effort? Are you planing to open a pull request any time soon or should I proceed with --cpp-namespace override pull request?

Thanks

[mikemccarty-vertex]

@zats I have a version that works (or appears to) across the supported languages in the sense that it generates reasonable compilable code. Like you, I'm not comfortable enough with all of the languages to say it officially works for all of them -- C++ and Java were my main focus.

In any case, as I said above, once I discovered that my changes broke gRPC ABI compatibility across languages, I essentially abandoned it because that is a critical use case for me and I don't know that the problem is fixable. If this feature is ever incorporated into mainstream flatc, it will have to come with a big gRPC disclaimer.

I would say, if you have something that is ready to ship right now, you should probably submit it. The fork I have is several months behind the head and I don't have the time to maintain it at the moment. If you want to incorporate some of the changes I have made into your implementation, I have no problem with that. The changes I made are not that sophisticated. ;)

[stefansullivan]

@zats @mikemccarty-vertex I can attest to the fact that the fork for --namespace-prefix works for a non-gRPC use case. I'm struggling with a python bug with importing from nested namesoaces that doesn't appear to come from this branch. But I'm guessing with a different schema structure it would be fine

[stale]

This issue has been automatically marked as stale because it has not had activity for 1 year. It will be automatically closed if no further activity occurs. To keep it open, simply post a new comment. Maintainers will re-open on new activity. Thank you for your contributions.

[WilliamWhispell]

I want to create a c++ service that is capable of returning either flatbuffer or protobuf. I'm able to convert all of my protos to fbs and then generate the code... the problem is: then the generated code is in the same namespace as the protobuf generated code so I can't compile with both headers included in the same library.

We need a way to be able to change the namespace that the generated code lives in and still provide cross language compatibility. It's great that Rust users can change the namespace, but it looks like c++ users are forced to search and replace package names in the proto files before converting to fbs and then generating the code...

[aardappel]

@WilliamWhispell how about replacing the namespace in the generated fbs before generating C++ code?

I suppose we could add a way to replace namespaces for --proto mode, but it seems a bit special purpose to me.

[WilliamWhispell]

@aardappel - At first I went the path of replacing namespace in the generated fbs, it's doable, but feels hacky as its a search and replace in a custom script. Basically same thing I'm doing by replacing the package name in the protos before generating the fbs.

I care specifically about using proto as the source of truth for our services. If we look at a google package, google.protobuf, at the any.proto file: https://raw.githubusercontent.com/protocolbuffers/protobuf/master/src/google/protobuf/any.proto - If we convert this to fbs, then generate the code, it will be in the namespace google.protobuf, which doesn't really reflect the structs that live there, because they aren't protos, they're flat buffers.

I think the flatc executable should allow you to change the package name when converting from proto to fbs. This lets you correct misnomers and has the added benefit that people can then use code generated from protos and flatbuffers in the same project without needing to do a custom search and replace.

[aardappel]

Hmm, I guess we can add it. I'd be in favor of it being a generic namespace replacer though, such that it can also be used with other outputs.

--replace-namespace A B ?

[WilliamWhispell]

Looks good to me.

[4ntoine]

We've jumped into "flatbuffers" name collision in Rust if using "something.flatbuffers" in schema namespace:

error[E0260]: the name flatbuffers is defined multiple times

I've found there is an option to override Golang namespace, is there anything language-agnostic?

[bmhatfield]

Hi there, a Go person chiming in here. There appears to be a misunderstanding about go package naming vs import paths.

When I generate code, I only need flatbuffers to generate the package name, and I can set output prefixes of where the packages should live on disk. The import path (ie, github.com/myrepo/packagename) is only particularly relevant at import time. The code generator doesn't really need to know about it at all, assuming I can place the files in an appropriate place on disk (ie, a package output prefix path).

It would be really nice if we could specify language-specific overrides in the schema files themselves, akin to how protobuf supports this. There's lots of language-specific wrangling that has to be done.

[stefansjs]

Hey folks, sorry to resurrect such an old issue. I have been using the --namespace-prefix branch since I last commented. I'm trying to merge in v2.0 but it's not a trivial merge. It sounded like we had a bit of support to make this into a feature in the official releases, but we haven't touched it in a long time. It's there any chance of pulling this into the Google repo and supporting it officially?

[aardappel]

@stefansjs someone would have to make a PR for it.