[RFC] Workspace generation

[sbarow]

I wanted to open the discussion around workspace generation and dependencies - opening a new task for the issue for a more concrete discussion.

To support making applications as modular as possible we would want a app/project.yml file to be able to specify a dependency on another lib/project.yml file. Looking at the way buck refers to dependencies in the BUCK file (bazel does something similar bazel ) they go with the approach of //<relative path>:<project-name>. Doing this would require users to generate their projects from the root of their source code and passing in the path to XcodeGen.

As an example lets assume we have a folder structure like this.

apps/
  app_1/
    ...
    dependencies:
      - //libs/libs_1
      - //libs/libs_3
    ...
  app_2/
    ...
    dependencies:
      - //libs/libs_2
    ...
libs/
  lib_1/
    ...
    dependencies:
      - //libs/libs_2
    ...
  lib_2/
    ...
    ...
  lib_3/
    ...
    ...
Carthage/
  ...

We could achieve this by introducing a new project dependency type which takes in a string prefixed with // and generating an .xcodeproject per project.yml and all of its dependencies.
Running XcodeGen //apps/app_1 would generate a workspace for app_1 at apps/app_1/App_1.xcworkspace as well as .xcproject in the desired project folder and all dependency folders. The workspace structure would mimic the folder structure defined in the directory.

• .xcodeproject - having nested projects might break things or make it more complex.
• Framework linking will be done through implicit dependencies in the .xcworkspace

Questions:

• Is this something XcodeGen should/wants to be able to do?
• Is this approach too opinionated?

[rahul-malik]

@sbarow - This is something we want as well and it seems like handling the generation of multiple projects and managing their build dependencies should be in the scope of XcodeGen. We're using Bazel over at Pinterest so I'm a bit biased in favor of your exact implementation suggestion :)

• I imagined a workspace feature would largely be additions into the project specification (introduce a top-level Workspace object) that would be able to reference a series of file paths to other yaml files that contain full project descriptions. In addition it would probably contain other metadata around workspace name, settings, etc.
• Dependency targets could use the Buck/Bazel rule format or potentially something minimal like a tuple of (Path, Dependency).
• This RFC should handle dependencies that are not Frameworks so linking via "find implicit dependencies" may not be sufficient so static libraries or other target types.

Thoughts?

[jerrymarino]

First off, this would be awesome 🥂

A couple cases I've been thinking about are:

• How propagating information across the projects would work. By transitive info, I mean build data that is not implemented as a target: linker flags, header search paths, etc.

• Xcode seems to not allow depending on files that aren't in the current project or a child of that project. So another edge case is copying over resources like .bundles, or .png's provided in by an adjacent project/dep. Ex: Carthage/SomeLib/SomeLib.bundle needs to end up in an adjacent and dependent iOS appApp/iOS/SomeApp.app.. I spent some time looking at how CocoaPods is doing this, and they codegen shell scripts for copy file phase, etc.

• Ability to support dependencies across projects #124

Also, as @rahul-malik mentioned, multiple projects is key for a larger app. I'd be interested in helping move this effort forward where I can!

[pepicrft]

Yeah! I think it'd be awesome to have such feature in XcodeGen 👏

I think it'd help a lot of companies with building modular apps. We at SoundCloud found very cumbersome to maintain a modular setup with multiple projects and workspaces. Buck & Bazel were options that we considered to do the automation but we didn't have enough resources to change our build infrastructure.

The way I imagined this automation as a developer working on a modular app was:

• I'm a developer that works on an app that depends on framework B, that depends on framework C.
• If I want to work on B, I could just write something like xcodegen generate-b and I would get a workspace that contains B and C and the necessary targets and schemes to build my module and its dependencies.
• If I want to work on the app, I could just write xcode generate-app and I'd get the whole tree generated.

Implementation-wise, I like @rahul-malik's approach. I think the current specification format can be extended to support as dependencies projects defined in other specification files. Do you think we would need the concept of Workspace in the project.yml file? When the project gets generated, we might need Xcode workspaces to include multiple projects that depend on each other, but do we need to define such workspace in the project.yml?

@rahul-malik could expand this point a bit?

This RFC should handle dependencies that are not Frameworks so linking via "find implicit dependencies" may not be sufficient so static libraries or other target types.

[sbarow]

@rahul-malik on your points:

1. I like the idea of a workspace object for naming, settings etc. Something we do within our BUCK file is to specify "workspace schemes" which can be a variant of the target with different schemes to include which is great if you have a project with hundreds of modules/projects. Regarding the paths to other .yml files, I would argue that this can be done through the dependency reference i.e //libraries/lib_1 which would generate that dependency and any dependencies it has.
2. Does yaml support tuples?
3. Handling of dependencies should be done through the standard project.yml dependencies spec right?

This RFC should handle dependencies that are not Frameworks so linking via "find implicit dependencies" may not be sufficient so static libraries or other target types.

I probably worded this incorrectly. It would only implicitly link built frameworks of projects within all of the .xcodeproj within the workspace.

@jerrymarino on your points:

1. Good point. This can be solved with something like include_defs we do something similar with BUCK
2. Those bundles would be included in the built product right? If so this is more an implementation detail for the user? The user should be specifying they want X resource in Y bundle i.e Bundle.url(forResource name: String?, withExtension ext: String?) -> URL?
3. This sounds like a cyclic dependency and something we should try guard against.

Regarding multiple projects, this is what I have working so far (includes Carthage deps)

@pepibumur to all your points, this is what I have working currently ☺️

[jerrymarino]

@sbarow awesome work!

I think the answers to the first 2 points are incredibly opinionated:

• P1: Include defs is a useful ability in Buck. In Bazel some metadata is propagated by the build system to compiler invocations automatically. i.e. a dependencies include's are propagated to dependents.

• P2: Bazel is automatically propagating bundle dependencies to the App from all objc_library and dependencies. For example, a Pod under Bazel 's bundle is automatically added to the output App without any configuration on the user's end.

The above points, can be handled today in today's project spec, but it is not trivial especially with a large number vendor dependencies 🐉

Maybe the answer is more meta: Does XcodeGen have opinionated logic that makes it easier for people to use? Or is it a higher level xcodeproj spec.. Both have trade offs.

• P3: Some dependencies may be shared dependencies of their dependencies. This already works in Xcode as long as the graph is acyclic, ex:

target: App
     deps: A
target: B
    deps: C, D
target: C
   deps: D
target: D
....

[rahul-malik]

@pepibumur - My thought on the implicit framework linkage was purely just mentioning we need to keep other output products like static libraries in mind when figuring out how to build projects where a dependency might exist in a different project within the same workspace.

@sbarow:

... Regarding the paths to other .yml files, I would argue that this can be done through the dependency reference...

• I don't feel too strongly about file-paths to other yaml files but it seems like a simple extension for users that are unfamiliar with Buck/Bazel. I think it's easy enough to learn though and this is more of a power-user feature IMO

Does yaml support tuples?

• YAML doesn't support tuples afaik so we'd have to have a less elegant nested array to simulate it

Handling of dependencies should be done through the standard project.yml dependencies spec right?

• Yes

[rahul-malik]

@sbarow @pepibumur - How do you suggest moving forward? I think between all of us we have similar app architecture needs (modularization, multi-project, etc.) so I believe we can find a good solution that works for all of us.

If i'm understanding the RFC correctly:

• It seems like we would have many project.yml files at various levels of the file hierarchy
• xcodegen would be modified to take a list of targets you'd like to build. Those targets and their transitive dependencies would be generated in one oiplr more projects based on the number of project.yml files that are referenced
• Specifications would have to be called project.yml since that is not a part of the dependency format currently and we would have to rely on convention.

These points seem reasonable to me. In addition I'd like to suggest:

• Any references from a Target to another Target (dependencies, testTargets, etc.) are able to be specified with the //path/to/target/directory:target syntax.
• Like Bazel/Buck, all rule target references are relative from the project root (no "../dir:target")
• We should establish a method of identifying the project root directory. This could be the location of SettingsPreset directory or potentially another file. This would allow us to perform generation from anywhere in the repository.
• Generated project names will have to be somewhat unique in case we are generating multiple projects that transitively reference the same project.yml files. This seems like it would happen often with your core libraries.
• For debugging / testing, we should add ways to print the dependency graph of what we're generating. Ideally the graph would be printed in a stable ordering for snapshot tests.
• For generation performance, we should design this in a way where we can parallelize the generation code in XcodeGen since it's entirely single-threaded right now and depending on the number of projects we can have significant wins by generating / writing them concurrently. A simple approach might be to split up the analysis of all the projects (and which targets) we are generating and then dispatching the individual project generation on dispatch queues.

Thoughts? Would also love to know how we can break down this feature so we can all contribute and help out :)

[pepicrft]

Yeah, this is a need may big apps have, I agree with moving this forward. What do you think about moving this RFC to a Google Docs? It'll be easier to leave comments there and think about how to break this down to do it achievable steps.

[sbarow]

Before opening this RFC, I worked on a version of this in Python. It might be useful to have a look at workspacegen

I am happy to add you guys to this, we could use the readme as a kind of RFC/TODO list if that makes sense? Google Doc can work also.

I'll work on this a bit more tomorrow and get some thoughts down on the points raised above.

[rahul-malik]

@sbarow - Thanks for sharing! Will take a look soon.

@pepibumur @sbarow - I think there is value with getting an RFC doc started and I'm happy to put one together based on this conversation.

One approach I thought of is that we might be able to approach this in a few phases:

1. Any necessary schema changes and xcproj changes necessary to create / modify a workspace
2. Adding simple query-like functionality to report things like "dependencies of a target", etc which will be helpful for this tool and others that would like to analyze their build configuration.
3. Adding/updating a command line argument like the one described originally for creating a workspace based on one or more rules. We could leverage the work in the above tasks to generate a project.yml that reflects the subset of targets you are generating which will allow us or anyone to debug issues.

[pepicrft]

Sounds good, would you mind sharing the one that you already started @sbarow? I think we can move from there and come up with implementation phases that we can work on, and maybe do things in parallel if it's possible.

Adding to your points @rahul-malik:

1. Yeah, right now xcproj scheme support is very basic. There's work we need to do there to support this work.
2. That would be great. We could actually generate a graphviz file to visualize the dependencies (I find it very hard to do it with Xcode nowadays)

[zdnk]

Any progress here?

[liamnichols]

I might be missing something, but it looks like more recent changes such as support for External Target References (#655) and soon to be support for nested subprojects (#701) solve the original discussions above?

Now it is more about just generating a workspace file? I feel like that can be quite a simple task that could become really powerful if we use this to bring caching and postGenCommand capabilities to a workspace level?

For example, being able to run a command something like the following:

$ mint run yonaskolb/XcodeGen xcodegen --workspace ./workspace.yml --use-cache

Where workspace.yml would look something like this:

name: MyWorkspace
options:
  postGenCommand: bundle exec pod install
settings:
  # WorkspaceSettings.xcsettings things would go here
projects:
  App:
    path: App
    spec: App/project.yml
  Frameworks:
    path: Frameworks
    spec: Frameworks/project.yml

I'm coming at this from the perspective where I don't really need XcodeGen to generate me a workspace since CocoaPods will do it however there is a benefit by adding support to XcodeGen since it'll allow us to utilise caching to only run postGenCommand more efficiently