Create an OLP Release

[oubiwann]

Epics:

• Create Supporting Infrastructure - Create Supporting Infrastructure #2
• Examine OTP and Follow as Example - Examine OTP and Follow as Example #8
• Create an Initial Build - Create an Initial Build #9
• Examine Release Use Cases - Examine Release Use Cases #36
• Standardize Release Testing - Standardize Release Testing #20
• Standardize Release Documentation - Standardize Release Documentation #28
• Support Multiple Releases - Support multiple releases #35
• Provide Multuple Means of OLP Installation - Provide Multuple Means of OLP Installation #37

[yurrriq]

@rvirding (or anyone), if you've got links to the change in direction of the OTP team regarding release structure/scope, please share.

[rvirding]

I have asked my colleague Lukas who works there and will get back as soon as I know.

[rvirding]

Slow down a bit until we know where we want to go. Do we want:

1. users download which then downloads everything for them (with rebar3 maybe, or whatever) and builds a release when they do make
2. we generate a release for them which they compile
3. we generate a pre-compiled release

Erlang/otp does 2 while elixir does 3, at least with brew.

[rvirding]

We can move the discussion from lfe to here.

[rvirding]

We should also discuss how we are going handle testing and documentation.

[oubiwann]

Believe it or not, this is actually moving slowly :-) It looks like a lot, since there is some infrastructure set up to explore this problem space ... but it's very early stages; this repo and Makefile don't do much yet.

What it does do right now:

• setup git submodules for LFE and all associated libraries (just ltest and lutil for now)
• run the rebar3 release command which builds an early version of a release

Current problems or things it doesn't do yet:

• there's no lfe executable in the release bin dir -- it's only in the LFE release dir ...
  • while most libraries will most likely keep their binaries in their own release directory bin subdir, I would think we'd want to bring the lfe executable up to the main dir
  • this is where erl ends up in a release, too -- isn't it?
• there's a useless olp binary getting created, and I don't think we want that
• no gzipped tarball is being created yet

To answer your release questions:

• This repo is intended for us to use more than anyone else, for the purpose of creating OLP releases
• It could, of course, be used by anyone who wanted to build an OLP release of their own
• As such, I believe this would move us in the direction of either option 2 or 3 in your list above -- I hadn't thought about that bit very much yet, but I think I would tend to go for option 2 right now ... lighter burden on us and we'd be following in the footsteps of the OTP giants ;-)

As for testing ... is there a make target for running the LFE tests yet? My thoughts on this would be to do something very simple:

• provide a make target at the OLP top level
• this wouldrun LFE's make test target
• it would run rebar eunit for each library in the lib dir

This, of course, would be used for us as a pre-build sanity check. Post-build, we could do something similar that would assume the presence of our toolchain.

However, testing a release by a user who has downloaded the release ... I don't have a quick answer for that ... and really, this would be the most important bit of testing. It's also the sort of testing we'd want to do on Travis CI, once the release is created ...

[oubiwann]

Ah, and documentation ... I'll talk out loud to see where my thoughts on this are:

Well, crap ... I have no idea.

• For LFE, we've been installing man pages to the system ... but that's not an option when building a release -- everything needs to be self-contained
  • OTP generates web pages for its docs ...
  • I guess we could do something similar; pandoc supports HTML output
  • other thoughts?
• For the libraries, we've been running lodox ... it might make sense to keep doing that
  • how do we provide a unified experience for all libs in the OLP release?
  • how do we/can we unify the LFE docs with the lib/* docs?

[yurrriq]

Quick thoughts from my mobile:

• Pace: 👍
• Travis CI: We can take some cues from Idris
• Docs: Somewhere I have an experimental fork of lodox that uses the EDoc format internally, with that finished we could conceivably generate Erlang docs (e.g. LFE proper) as well as LFE docs. If that seems valuable, I'm happy to continue pursuing it. (Use EDoc format internally lfe-rebar3/lodox#30)
• Release delivery: As of now, I think I'm leaning toward supporting method 2 and encouraging method 3 from @rvirding's comment. That's basically how Idris does it. I'm happy to maintain the Nix package too.

[oubiwann]

For @rvirding's testing questions, I've created epic #20. I've updated this release ticket (#1) with a bullet item for the new epic.

Feature #23 tracks what I suspect is really the essence of his concerns (providing users without our dev tool chain the means of testing a release after they download it).

[oubiwann]

For @rvirding's docs questions, I've created epic #28 which provides a bullet list of the set of features I'm guessing we'll need to implement in order to address that. I've updated this release ticket (#1) with a bullet item for the new epic,

[oubiwann]

@yurrriq, my gut on adding EDoc support to lodox + 1

• it would be nice to provide an API reference for LFE as well as everything else we have
• ultimately, how useful this would be would be determined by how highly commented the LFE codebase is ...
• or, more optimistically, how highly commented we make it :-)

[oubiwann]

Okay, tickets galore created and updated. Back to some nitty-gritty stuff.

As I get time today, I'm going to try to work on basic release-generating tasks ... and spend a lot of time looking at how various libraries included in OTP handle their own release needs (executables, docs, etc.).

[oubiwann]

Big progress -- just confirmed LFE and Erlang work in a standalone install of the relase, with full access to LFE libraries, etc.

[rvirding]

I am still confused. :-) Reading through the README and the issues I get the impression that OLP is something the end LFE users clone olp and then use it to build their own LFE. This is not how OTP does it. The otp repo is not meant to be downloaded by users but it is used to build a release which is then distributed to users who can them build their Erlang/otp with it. So much of what is in the otp repo never reaches end users.

This is how I think we should do it as well.

If this is what is intended then I have misunderstood it. Sorry.

P.S. I am not that much a rebar3 fan as by default it tends to put things in strange places, the _build/... tree, instead of where things "normally" are in a release. But maybe if we are using it to build a release which we then distribute it will work.

[rvirding]

P.P.S. I will not have that much time in the coming months to work on this. For good reasons. Currently I am working on an "LFE for Erlangers" course/tutorial which we will give internally next week. Then I have a little project for a client on translating C# code to LFE. Don't ask. Then in dec/jan I will be away for about 4 weeks travelling on holiday and I won't get much chance to work on LFE. Unless of course I can run it all on a windows machine and convince the family that it is not work. :-)

[oubiwann]

Hey Robert!

Anticipated OLP users, type of usage, and what they'd be using:

• Devs working on their own LFE projects, who want all batteries included, using the OLP release (prebuilt)
• Devs working on their own LFE projects, who want all batteries included, using the OLP release (want to build it themselves)
• Maintainers, who are using this repo to create releases, using the make targets, rebar3, etc.

Of those, I would expect the highest number of users to be the first group. Related: there is a ticket that's been created to explore the possible use cases and to make sure OLP meets anticipated needs reasonably and well. Here's the ticket.

The reason that the README is confusing you is that currently, the only people using OLP are us :-) The instructions you see in the README at the end are "For OLP Developers" ... once other use cases are examined and a clear method has been devised for supporting them, the README will be updated.

I think we should definitely follow the spirit of OTP as much as possible, with our ultimate aim being to support the users of that first bullet. In other words, I totally agree with you :-)

As for rebar3, I do happen to be a big fan (it has made my life so much easier in so many ways ...) -- but! No need to worry -- the rebar3 usage in this repo isn't causing us to do anything that goes against the spirit of OTP -- and we won't be asking users to wrap their heads around anything it does, either. Though the release is generated in _build/default/rel, that's not how we'll present it to users:

• In the OLP docker image I'm building, the release is put in /opt/olp/
• If we later provide downloads, we'll simply be giving folks the tarball that the release creates
• For the odd OLP user that wants to build OLP themselves, we'll provide a make target that does something like what the docker image does: install the release to a location like /opt/olp

Put more simply, your guess was correct :-) That's how we're using rebar3, simply as part of the OLP build toolchain.

Enjoy your vacation! And hooray for the tutorial! (And sorry about the C# ...)

[oubiwann]

@rvirding To assist with the confusing wording in the README, I've made these changes:

• Changed the section "For OLP Developers" to "For OLP Maintainers"
• Added a new section (with a TBD comment) for "For LFE/OLP Application Developers"

There's no sense in tweaking the other sections right now (docker, build, etc.), since that's going to radically change once we hammer out the use cases.

Hopefully the "Here there be Dragons" warning at the top doesn't let anyone get to high of an expectation right now, since this is a new project in a high state of flux and early development ...

[rvirding]

I have talked with the OTP team about their splitting up the OTP release into multiple packages. It is coming. In 20 they plan to split out some of the least used applications, while in 21 to the whole system into multiple packages where the core one would basically be the current Basic application group (what you need to run the system).

Rebar3 is THE tool for the system which they plan to use to handle the packages. But they will get rebar3 to build the packages with the current structure for backwards compatibility. Apparently they will be able to fix this with the right rebar.config. Something for us to steal.

[yurrriq]

Related: lfe/lfe#303

That PR currently sets up this structure:

/usr/local/lib/lfe
├── bin
│   ├── lfe
│   ├── lfec
│   ├── lfedoc
│   └── lfescript
└── ebin
    ├── cl.beam
    ├── clj.beam
    ├── lfe.app
    ├── lfe.beam
    ├ # ... elided for brevity
    └── lfescript.beam

2 directories, 35 files

but I like the Erlang/OTP structure for OLP:

/usr/local/lib/erlang
├── Install
├── bin
│   ├── ct_run
│   ├── dialyzer
│   ├── epmd -> ../erts-8.1/bin/epmd
│   ├── erl
│   ├── erlc
│   ├ # ... elided for brevity
│   └── typer
├── erts-8.1
│   ├── bin
│   ├── include
│   ├── lib
│   └── src
├── lib
│   ├── asn1-4.0.4
│   ├── common_test-1.12.3
│   ├── compiler-7.0.2
│   ├── cosEvent-2.2.1
│   ├ # ... elided for brevity
│   └── xmerl-1.3.12
├── misc
│   ├── format_man_pages
│   └── makewhatis
├── releases
│   ├── 19
│   ├── RELEASES
│   ├── RELEASES.src
│   └── start_erl.data
└── usr
    ├── include
    └── lib

61 directories, 22 files

[rvirding]

Yes, we should definitely structure it as an OTP release with a lib directory for the all the applications. Then lfe will just be one of the applications together with the other "standard" applications like ltest. Then you can at build time control whether each application should just consist of the obligatory bin and priv directories or whether you want to include say src and include as well. Even if everything is precompiled some applications should probably include the include directory as well, for example lfe itself and flavors.

[oubiwann]

Yup, just to confirm, this is what's already being done in OLP. Following OTP as much as possible :-)