Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make sure Puma has restarted properly #4127

Merged
merged 1 commit into from
Sep 22, 2020

Conversation

javierm
Copy link
Member

@javierm javierm commented Sep 9, 2020

Background

We were getting an error when restarting Puma after upgrading Ruby. Even if the restart command was sent successfully, Puma silently crashed and the log had the following error:

/home/deploy/.rvm/rubies/ruby-2.4.9/lib/ruby/site_ruby/2.4.0/bundler/spec_set.rb:91:in
`block in materialize': Could not find rake-13.0.1 in any of the sources
(Bundler::GemNotFound)

So it looks like the crash happens because Puma was started when the application used Ruby 2.4 and now when it's restarted it still tries to use Ruby 2.4, even if the application now uses Ruby 2.5.

Objectives

  • Make sure Puma is started if it crashes after restarting it
  • Avoid getting a 502 error after upgrading to version 1.2

Notes

I haven't found a proper way to configure Puma so we can avoid this, so as a workaround I've added the puma:start task after restarting Puma. If Puma was successfully restarted, puma:start will do nothing; if Puma crashed, puma:start will start it.

To guarantee the tasks will be executed in the proper order, the tasks introduced by capistrano3-delayed_job and capistrano3-puma are cleared, and then we configure the order so first we restart Puma, then restart the Delayed Jobs processes (so there's enough time for Puma to crash if Ruby was upgraded) and then start Puma.

@javierm javierm added this to Reviewing in Consul Democracy via automation Sep 9, 2020
@javierm javierm self-assigned this Sep 9, 2020
We were getting an error when restarting Puma after upgrading Ruby. Even
if the restart command was sent successfully, Puma silently crashed and
the log had the following error:

/home/deploy/.rvm/rubies/ruby-2.4.9/lib/ruby/site_ruby/2.4.0/bundler/spec_set.rb:91:in
`block in materialize': Could not find rake-13.0.1 in any of the sources
(Bundler::GemNotFound)

So it looks like the crash happens because Puma was started when the
application used Ruby 2.4 and now when it's restarted it still tries to
use Ruby 2.4, even if the application now uses Ruby 2.5.

I haven't found a proper way to configure Puma so we can avoid this, so
as a workaround I've added the `puma:start` task after restarting Puma.
If Puma was successfully restarted, `puma:start` will do nothing; if
Puma crashed, `puma:start` will start it.

To guarantee the tasks will be executed in the proper order, the tasks
introduced by capistrano3-delayed_job and capistrano3-puma are cleared,
and then we configure the order so first we restart Puma, then restart
the Delayed Jobs processes (so there's enough time for Puma to crash if
Ruby was upgraded) and then start Puma.
@javierm javierm force-pushed the puma_start_after_upgrading_ruby branch from 13c8ceb to 40b3c9f Compare September 16, 2020 08:53
Copy link
Member

@Senen Senen left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Production tests were successul! 👏

Consul Democracy automation moved this from Reviewing to Testing Sep 22, 2020
@javierm javierm merged commit 6c0fc33 into master Sep 22, 2020
Consul Democracy automation moved this from Testing to Release 1.2.0 Sep 22, 2020
@javierm javierm deleted the puma_start_after_upgrading_ruby branch September 22, 2020 11:49
@javierm javierm removed the 1.2 label Sep 22, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants