Add manage_service settings to get puppet out of the way.

[jguiditta]

This should make starting services more consistent, as pacemaker will have full control of starting and stopping the services without the chance of puppet having already done so, which potentially
could cause confusion for pacemaker.

[jguiditta]

This still has a problem or two to be worked out, specifically errors against rabbitmq

[cwolferh]

This contain, along with contain cinder::scheduler, caused a dependency cycle error like

Error: Could not apply complete catalog: Found 1 dependency cycle:
(Cinder_config[DEFAULT/glance_api_version] => Service[cinder-api] => Class[Cinder::Api] => Class[Quickstack::Cinder] => Class[Quickstack::Cinder_volume] => Cinder_config[DEFAULT/glance_api_version])

when the paramter $backend_rbd in quickstack::pacemaker::cinder was set to true

Removing the 2 contain's caused the dep cycle error to go away. However, I had to add 2 extra dependencies to make sure that the Service's were started before we executed the one-time stop and disable. See
https://github.com/cwolferh/astapor/compare/jguiditta:add_manage_service_ha...cwolferh:service_unmanage_tinkering?expand=1

[jguiditta]

Results of successful run:

• pcs status - https://gist.github.com/jguiditta/6ac02293087eeec9e18f
• openstack-status from node 1 (horizon always shows as uncontactable with this tool in HA, since it is not bound to localhost) - https://gist.github.com/jguiditta/23a7e3a09ae8171a63f1
• node 2 - https://gist.github.com/jguiditta/97447ffe1415cae67e06
• node 3 - https://gist.github.com/jguiditta/6542d9af07f25ea6fa01

Note that services that should be A/P like neutron (anything but server) and heat engine, are shown as active on only one node, and inactive on the others.

[cwolferh]

Looks good, services have been disabled: http:https://ur1.ca/i4jhy

[cwolferh]

pacemaker did not bring up rabbitmq across all nodes after a couple of fresh attempts. E.g.:

# pcs status
Clone Set: rabbitmq-server-clone [rabbitmq-server]
Started: [ c1a1.example.com c1a2.example.com ]
Stopped: [ c1a3.example.com ]

From the puppet output on the control node:

Debug: Execone-time-rabbitmq-server-disable: Executing '/sbin/chkconfig rabbitmq-server off'
Debug: Executing '/sbin/chkconfig rabbitmq-server off'
Notice: /Stage[main]/Quickstack::Pacemaker::Rabbitmq/Quickstack::Pacemaker::Manual_service[rabbitmq-server]
/Exec[one-time-rabbitmq-server-disable]/returns: executed successfully
Debug: /Stage[main]/Quickstack::Pacemaker::Rabbitmq/Quickstack::Pacemaker::Manual_service[rabbitmq-server]
/Exec[one-time-rabbitmq-server-disable]: The container Quickstack::Pacemaker::Manual_service[rabbitmq-server] will propagate my refresh event
Debug: Quickstack::Pacemaker::Manual_service[rabbitmq-server]: The container
Class[Quickstack::Pacemaker::Rabbitmq] will propagate my refresh event
Debug: /usr/sbin/pcs resource show rabbitmq-server > /dev/null 2>&1
Debug: /usr/sbin/pcs resource create rabbitmq-server systemd:rabbitmq-server op monitor interval=30s start-delay=35s interval=30s --clone
Error: Unable to create resource/fence device
Call cib_create failed (-206): Application of an update diff failed

If I comment out the above 3 lines quickstack::pacemaker::manual_service { "rabbitmq-server":..., rabbitmq comes up fine for me.

Clone Set: rabbitmq-server-clone [rabbitmq-server]
   Started: [ c1a1.example.com c1a2.example.com c1a3.example.com ]

The cause isn't clear to me, but I've got the logs handy.

[cwolferh]

I had success throwing in a "sleep 60" after manual_service { "rabbitmq-server": . Not that I'm suggesting that as the final solution, but perhaps we need to let systemd to finish reloading (if applicable) across nodes before a pacemaker resource is added.

[jguiditta]

Hmm, that is odd, since all we are doing is a chkconfig off, not even stopping the service. I wonder if it would be better to perhaps move the disable to happen at the very end of the first run?

[cwolferh]

Looks good. Services disabled after 2nd puppet run.