Stack level to deep error after upgrading to 4.4.0 #1427
Comments
|
I just checked Consider |
|
I think changing the timing of prepend could also solve the problem without introducing another patching method. I've implemented it in #1432 and will cut a release soon. |
|
|
|
I was hit by similar problem with redis instrumentation between opentelemetry and scout. they have migrated to using prepend in redis, but this issue tells me they should do it everywhere... |
|
FYI: This is very much not fixed in Not sure how much it will help, but see attached log. |
|
@mvastola I'm really sorry that it still causes you the problem 😔 |
|
Same issue: |
|
@RobWu @mvastola are you apps using Rails? It'd be helpful to know this information because I think I've found the cause. If it's not a Rails app, can you make this change: make sure If it is a Rails app, can you provide me the Rails version + your scout/sentry configs? thanks |
|
@st0012, see below. Part of the insidiousness of this though is that this issue only emerged in prod. (I think it was compounded by the fact that errors happen relatively more frequently in prod, and a stack overflow maxes out memory, so everything else either takes forever or fails.) We don't manually
if defined?(Sentry)
Sentry.init do |config|
config.dsn = Rails.application.credentials.sentry[:ruby_dsn]
config.breadcrumbs_logger = [:active_support_logger]
config.environment = Rails.env
# config.backtrace_cleanup_callback = lambda { |backtrace| Rails.backtrace_cleaner.clean(backtrace) }
config.release = File.read('RELEASE_VERSION').chomp
end
end
common: &defaults
key: [REMOVED]
monitor: true
ignore:
- '/webhooks/zendesk'
- '/404'
production:
<<: *defaults
auto_instruments: true |
|
@mvastola thanks for sharing those info. unfortunately, I still can't reproduce the problem with the same configs. common: &defaults
key: [REMOVED]
monitor: true
ignore:
- '/webhooks/zendesk'
- '/404'
- production:
+ development:
<<: *defaults
auto_instruments: truebtw, I don't recommend initializing the SDK conditionally. you can use the Sentry.init do |config|
config.dsn = Rails.application.credentials.sentry[:ruby_dsn]
config.breadcrumbs_logger = [:active_support_logger]
config.environment = Rails.env
config.enabled_environments = %w[production]
# config.backtrace_cleanup_callback = lambda { |backtrace| Rails.backtrace_cleaner.clean(backtrace) }
config.release = File.read('RELEASE_VERSION').chomp
end |
|
There is an open PR on the scout_apm gem which has flip the Net/HTTP instrumentation to use Module#prepend, I would try that next. |
# config/scout_apm.yml
common: &defaults
disabled_instruments:
- "NetHttp"solves the issue for me My config: Sentry.init do |config|
config.dsn = Rails.application.credentials[:sentry][:dsn]
config.enabled_environments = %w[production staging]
config.breadcrumbs_logger = [:sentry_logger]
config.excluded_exceptions = []
end |
|
So this is still very much broken despite upgrading to the latest versions of Our latest backtrace is here. I turned on some debugging information, which might make things easier to trace. I was finally able to reproduce the problem on a non-production instance, and -- trying to debug this a bit -- it looks like the execution flow is this: The irb(main):004:0> Net::HTTP.instance_method(:request)
=> #<UnboundMethod: Net::HTTP(Sentry::Net::HTTP)#request(req, body=..., &block) /usr/local/bundle/gems/sentry-ruby-core-4.5.1/lib/sentry/net/http.rb:35>.. and that irb(main):006:0> Net::HTTP.instance_method(:request).super_method
=> #<UnboundMethod: Net::HTTP#request(request_with_scout_instruments)(*args, &block) /usr/local/bundle/gems/scout_apm-4.1.1/lib/scout_apm/instruments/net_http.rb:28>.. that method, irb(main):003:0> Net::HTTP.instance_method(:request_without_scout_instruments)
=> #<UnboundMethod: Net::HTTP(Sentry::Net::HTTP)#request_without_scout_instruments(request)(req, body=..., &block) /usr/local/bundle/gems/sentry-ruby-core-4.5.1/lib/sentry/net/http.rb:35>.. thereby creating the infinite loop. I am not able to reproduce on a brand new rails app though, so I imagine the issue could just be with order in which these hooks are applied. |
|
Update:
|
* Add Hub#with_background_worker_disabled This helper temporarily disables background event dispatching when executing the block. * Wrap rake task's execution with Hub#with_background_worker_disabled This makes sure all events, whether manually triggered or handled by the SDK, can be sent synchronously. * Update changelog
|
@st0012 good catch. Thanks. Just not sure when "pretty quick" is since that comment you linked is from May. 🙁 |
|
@mvastola 4.1.1 solved the issues I had, so on my end, I'm Ok with the current state of things |
|
@st0012 I ran into the same issue due to the gem SystemStackError(stack level too deep) reproduction script:require 'bundler/inline'
gemfile do
source 'https://rubygems.org' do
gem 'rails', require: true
gem 'sentry-ruby'
gem 'skylight', '< 5.0.0' # SystemStackError
# gem 'skylight', '>= 5.0.0' # No SystemStackError
end
end
class TestApp < Rails::Application
Sentry.init do; end
end
TestApp.initialize!
uri = URI('http:https://github.com')
http = Net::HTTP.new(uri.host, uri.port)
http.set_debug_output($stdout)
http.request(Net::HTTP::Get.new(uri.request_uri))Update: Looks like catching the aliasing is not straight-forward. |
Same problem over here. Still having this issue with scout_apm (4.1.2) and Sentry (4.7.3). Problem also occurs with Sentry 4.4.2, 4.5.2 and 4.6.5. What is the recommended approach of action for now? Downgrade to Sentry 4.3 and wait for scout_apm 5 to land? |
|
@kreintjes I hope you (and others that still are bothered by the issue) to push Scout to migrate to the And before they finally do it, you can disable Scout's http instrumentation to workaround the issue: # config/scout_apm.yml
common: &defaults
disabled_instruments:
- "NetHttp" |
|
This issue has gone three weeks without activity. In another week, I will close it. But! If you comment or otherwise update it, I will reset the clock, and if you label it "A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀 |
|
Still tracking. |
|
What is the status on this? We had our app crashing in prod because of this issue. We had to disable scout entirely, because Sentry is more important on our stack. |
|
@jpaulomotta Sub-optimal, but a workaround to have both running since scout fix doesn't seem to be happening, at least this way (this is for sentry 5.1.0 with scout 5.1.1)
|
|
@jpaulomotta There's a glimmer of hope from Scout on this one but it's painfully slow going for some reason: scoutapp/scout_apm_ruby#448 |
|
Following @adi-pen's comment:
I saw Skylight's v5 announcement seems relevant:
Updating to Skylight v5 worked for us - required update to Ruby 2.7.3 however. (publiclab/mapknitter#1712) (error log had been this) |
|
Given that scoutapp/scout_apm_ruby#448 has been released in version If you have any similar issues, I recommend:
|
In the version
4.4.0, we added support for capturingnet/httprequests as span/breadcrumbs. This feature requires patching the class, which can be done in 2 different but conflicting ways: method aliasing and module prepending. Here's a great article from bearer explaining how these 2 approaches conflict with each other.Fortunately, the community has been adopting the
prependapproach for a while now +sentry-rubyrequires Ruby 2.4+. So the chance of having the issue should be low.How we're going to help you upgrade
With that being said, if you're reading this issue, chances are you're having the conflict inside your app. So here's our plan to help you upgrade:
prependapproach. Most of the gems that patch this library should have provided an option to switch the mode toprepend. For example,rack-mini-profilerand itsprepend_net_http_patchoption.opt-outthis patch.Update
The version
4.4.1should solve the issue.The text was updated successfully, but these errors were encountered: