Some questions on how to set up and upgrade a new platform, based on a default one

[santo74]

What's the recommended way to do the following ?

Set up a platform based on a default provided one, but with additional modules and still being able to easily upgrade this platform whenever a module or platform upgrade becomes available ?

E.g. I want to have a panopoly platform based on the one that gets installed by Octopus (i.e. panopoly with the additional optimizations).
But I need additional modules, such as multilingual capabilities and other contributed modules.

1. should I create a new platform for this, based on the default one or should I just use the default one and add the additional modules to it directly (with drush dl or similar) ?

2. In case of the former, then what's the recommended way to do this ?
   I can't find any makefile for the default panopoly platform installed by octopus (i.e. with optimizations)
   and I'm unable to create a makefile of it, as explained in this issue: unable to use drush make-generate #510

3. Depending on 1) and 2) above, how can I update the platform and/or the modules?
   Can I run octopus up-stable all and will it automatically take care of upgrading/migrating the existing platform, modules and sites on the platform ?

I'm not new to drupal, but I'm quite new to Aegir/Boa and still trying to understand how everything works and how things have to be done.

I'm aware of the nice documentation on the website of omega8.cc, but reading through those docs multiple times, I still ddin't find a complete answer to my questions above.

[omega8cc]

Try to keep this simple.

Basically, all your questions are answered in this section: https://omega8.cc/library/development

Just add your modules to the built-in platform: https://omega8.cc/how-to-customize-built-in-platforms-252

And if this is not enough, create custom platform: https://omega8.cc/how-to-add-custom-platform-properly-140

Also, please understand that Aegir helps you manage your sites upgrades, but it doesn't manage your codebases. It just uses the codebases - either built-in or custom you will create. Again, read https://omega8.cc/your-drupal-site-upgrade-safe-workflow-298 and other articles linked in https://omega8.cc/library/aegir-basics-index

That said, we could probably add a magic command like this:

$ octopus clone-platform o1 platformA platformB

[santo74]

Thanks for the info.
I agree that there are * general * answers to my questions in the docs on omega8.cc, but after reading them several times, I still wasn't fully satisfied and therefore asked the questions over here.

For example the drupal-site-upgrade-safe-workflow documentation gives a good explanation on how one can/should upgrade a site, but the first step says:

Create a new platform (or use some pre-built platform if available and applicable)

With links to the documentation on creating a new platform as well as a link to the "pre-built" platforms.
This is great, but doesn't explain how you can/should create a new platform for a non-core drupal platform, such as panopoly, commerce, etc

Based on the info in other docs I understand it's recommended to use a makefile for the platforms, which is probably perfectly doable for the core drupal platforms, as their make files can be easily downloaded from github. But for the non-core drupal platfroms this doesn't work because I can't just download a makefile an create a new platform off of it.
Therefore I tried to create a makefile from the existing panopoly platform with drush make-generate but this doesn't work, as explained in issue 510.

Another possibility would be to rsync everything from /data/disk/sat1/distro/001/panopoly-7.x-1.13-7.33.1 to /data/disk/o1/static but I'm not sure this is recommended ?
First of all it will copy over the symlink for the modules folder, which doesn't work when verifying the platform because of write permissions, but if I remove it and create a new folder with the correct owner/permissions, I lose the reference to the modules (while the verify task will succeed in this case)
Of course I can then manually copy over all the modules and change the owner/permissions, but then I'm not sure how I can update those modules in the future.

So as you can see, while the basics are answered in the docs, there are still some important gaps, especially for newcomers to the aegir/boa ecosystem, possibly because it requires a different way of thinking :-)

So yes, it would definitely be awesome if you could add the possibility to clone a platform.

Also, where can I find more documentation on the barracuda and octopus commands ?
And is there some more in-depth information on what an octopus up-stable exactly does, regarding updating platforms and such ?
Related to this, I just performed an "octopus up-stable all both" and everything went fine, except for one thing:"Aegir Hostmaster 002"
Before the update command there was only 1 hostmaster platform called "hostmaster" which contained the o1 site, but since the update there is an additional hostmaster platform called "Aegir Hostmaster 002" and now it's this platform that holds the o1 site.
Is this expected behaviour ?

I'm sorry for the lengthy post and the many questions, but I'm trying to understand everything as good as possible.
And of course I already read this doc as well https://omega8.cc/the-best-recipes-for-disaster-139

[omega8cc]

Every built-in distro comes with its own makefiles you can use to create just another copy of such platform in ~/static. Just modify that makefile to use our enhanced Drupal core. And that is it! Don't mess with creating makefiles from scratch. Just re-use what is already created. Maybe you need to learn more about drush makefiles. But it is outside of the BOA project scope. There are plenty of Drush related docs and articles on d.o

You can also rsync the platform locally (to skip the makefile build time) with -L flag to create a mirror copy with different name, and it will include complete code, with dereferenced o_contrib copied/embedded. Just remember to set correct permissions on platform root, /sites and /modules.

You can then delete that copied o_contrib directory and simply re-verify the platform in Aegir and it will instantly add the symlink to the system level o_contrib.

No magic involved.

BOA comes with built-in docs, and what is not included can be easily found on our website.

As for the hostmaster platforms, yes, that is how BOA and Aegir works.=

[santo74]

Ok, thanks, but then again, where am I supposed to find the makefiles ?
This is everything I can find on github:
https://github.com/omega8cc/boa/tree/master/aegir/makefiles
So nothing for panopoly, commerce, etc

[omega8cc]

We don't build these platforms from scratch, so we don't provide copies of their own makefiles. You can find them in the respective platforms codebases/profiles which are downloaded from their repositories on drupal.org

You can also download and use their already built codebases from drupal.org and only replace the Drupal core - that is exactly what BOA does. No need to run makefiles if you intend to start with some ready to download distro as a base for your custom platform.=

[santo74]

Allright, that's apparently part of the misconception then.
I thought there were also modifications/optimizations made to those platforms and not only to drupal-core.

So if I understand it correctly, I can for example set up an OpenAid platform by simply downloading their makefile and replacing the drupal core with the one from boa and in that way still benefit from all the optimizations made to boa ?

[omega8cc]

Correct. We don't modify any distro, with rare exceptions, if there is a known bug or duplicate module already provided by BOA specific o_contrib so it could generate PHP errors, so it need to be removed. Or if there is critical security upgrade to some contrib module not yet included in the distro. But these are rare exceptions. Otherwise they are just vanilla distros + our enhanced Drupal core + automatically symlinked o_contrib.=

[santo74]

Wonderful, this helps a lot in my understanding of the inner workings of Aegir/Boa!
Thanks for the quick and clear responses!

[omega8cc]

We should improve our docs, perhaps.

[kopeboy]

Yes you should, cause I am lost exactly like santo74 or even more.

I would add this too:
How to update a pre-built platform?
Where/how to download/use pre-built platforms when they are erased from my instance?
(I can only see the platforms I am already using because all the others have been deleted after 30 days..)
How to update my Aegir Hosting System to a new BOA version? (I see no posts about this on the omega8.cc site)

[omega8cc]

How to update a pre-built platform?

You can't and shouldn't "update" any built-in platform.

Please read for more information and context: https://omega8.cc/your-drupal-site-upgrade-safe-workflow-298

You can add/install/re-add new platforms by running octopus upgrade also when there is no new BOA version if old platforms have been deleted, but it is not possible on our hosted BOA - you need to submit support request then.

As for the BOA system and platforms upgrade - it is explained in this project docs, but it doesn't apply to hosted BOA, so it is not explained on our website, because we handle these upgrades as a part of managed hosting service.

[ghost]

I agree that updating the help docs would be good. Please add specific examples.