Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sentry timeouts don't work on our server #777

Closed
xevolic opened this issue Nov 10, 2023 · 8 comments
Closed

Sentry timeouts don't work on our server #777

xevolic opened this issue Nov 10, 2023 · 8 comments

Comments

@xevolic
Copy link

xevolic commented Nov 10, 2023

How do you use Sentry?

Self-hosted / on-premises

SDK version

4.5.0

Steps to reproduce

We have an application, which we develop in Docker environment.

We added three options to Sentry configuration in sentry.yaml file:

    options:
        attach_stacktrace: true
        send_attempts: '%env(int:SENTRY_RESEND_ATTEMPTS)%'
        http_connect_timeout: '%env(int:SENTRY_HTTP_CONNECT_TIMEOUT)%'
        http_timeout: '%env(int:SENTRY_HTTP_TIMEOUT)%'
        environment: '%kernel.environment%'
        error_types: 'E_ALL & ~E_NOTICE & ~E_DEPRECATED'
        release: '%kernel.environment%'
        integrations:
            - 'Sentry\Integration\IgnoreErrorsIntegration'

Our values are:

SENTRY_RESEND_ATTEMPTS=0
SENTRY_HTTP_CONNECT_TIMEOUT=1
SENTRY_HTTP_TIMEOUT=2

We tested these timeouts in Docker environment and all response times from our backend application were correct.

But when we deployed this application to our server they seems to be ignored.

We tested Sentry timeout with some page with sleep(10); to get response after 10 sec.

This approach worked in Docker, but does not work on the server: we need to wait these 10 seconds for Sentry, but it should be quit after 2 seconds (timeout limit reached).

Can you help us in finding what we are doing wrong?

Expected result

Sentry timeout should work as expected.

Actual result

We need to wait for e..g 10 seconds for page, which simulates Sentry.

@cleptric
Copy link
Member

cleptric commented Nov 10, 2023

The defaults are applied by the underlying PHP SDK here https://github.com/getsentry/sentry-php/blob/3.x/src/Options.php#L25-L35.
Hence I'm wondering why you see a 10 second timeout, it almost feels like that the env vars you set on production somehow set these options to null.

Another possibility could be that you're using an HTTP Client we do not "support", as in we don't apply the options here https://github.com/getsentry/sentry-php/blob/3.x/src/HttpClient/HttpClientFactory.php#L112-L164

@xevolic
Copy link
Author

xevolic commented Nov 10, 2023

@cleptric thanks for your reply

I forgot to mention that our application uses Symfony framework and uses following configuration file (sentry.yaml):

sentry:
    dsn: '%env(SENTRY_DSN)%'

#    If you are using Monolog, you also need these additional configuration and services to log the errors correctly:
#    https://docs.sentry.io/platforms/php/guides/symfony/#monolog-integration
    register_error_listener: false

    options:
        attach_stacktrace: true
        send_attempts: '%env(int:SENTRY_RESEND_ATTEMPTS)%'
        http_connect_timeout: '%env(int:SENTRY_HTTP_CONNECT_TIMEOUT)%'
        http_timeout: '%env(int:SENTRY_HTTP_TIMEOUT)%'
        environment: '%kernel.environment%'
        error_types: 'E_ALL & ~E_NOTICE & ~E_DEPRECATED'
        release: '%kernel.environment%'
        integrations:
            - 'Sentry\Integration\IgnoreErrorsIntegration'

We try to simulate 10-second delay of Sentry application to check if timeout values are working as expected.

Our application connects to SMTP server and if it rejects our request, an error is sent to Sentry.
In that case, with high traffic, we killed Sentry some time ago and want to prevent it from happening again.

We can increase 10-second delay to 15 or 30 seconds... we just chose 10 at the moment.

@cleptric
Copy link
Member

The Symfony SDK uses the PHP SDK under the hood https://github.com/getsentry/sentry-symfony/blob/master/composer.json#L31, so I just wanted to point out where the default values are coming from. I assume something is wrong with your production server, as the config values are correctly applied in my tests.

If you are generally concerned about the overhead the PHP SDK is introducing to your request times, I recommend running a local Relay next to your application https://docs.sentry.io/platforms/php/guides/symfony/performance/#improve-response-time.

This decreases the roundtrip time, and even if your on-premise instance is not accepting any events, the local Relay will still respond with a 200 status code.

@xevolic
Copy link
Author

xevolic commented Nov 15, 2023

We made a workaround by replacing default timeout values in vendor/sentry/sentry/src/Options.php (with sed), because setting timeouts in sentry.yaml didn't work for us.

@cleptric
Copy link
Member

You mentioned it was working in Docker but not in your production environment.
Did you try setting the raw values in your sentry.yaml instead of using environment variables?

@xevolic
Copy link
Author

xevolic commented Nov 17, 2023

Hi there.

Yes, I tried setting raw values in sentry.yaml, but they didn't work for me (they were not been applied on the server).

I'm not sure if it could help you, but when I was debugging the issue:

  • in file vendor/sentry/sentry/src/Client.php:
    • I got locally in Docker:
captureEvent() :: 2023-11-13T20:35:20.235179 :: start
captureEvent() :: 2023-11-13T20:35:22.240692 :: Operation timed out after 2001 milliseconds with 0 bytes received for "https://beta.somewebapp.pl/api//store/".
captureEvent() :: 2023-11-13T20:35:22.240692 :: Symfony\Component\HttpClient\Exception\TransportException
  • and on the server:
captureEvent() :: 2023-11-13T21:14:14.218842 :: start
captureEvent() :: 2023-11-13T21:14:16.222341 :: The promise was rejected
captureEvent() :: 2023-11-13T21:14:16.222341 :: GuzzleHttp\Promise\RejectionException

@cleptric
Copy link
Member

Please try to update the SDK composer update -W sentry/sentry-symfony as you're running a quite old version of the SDK.

@cleptric
Copy link
Member

cleptric commented Apr 8, 2024

Closing due to inactivity.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Archived in project
Development

No branches or pull requests

2 participants