Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implemented database reconnection loop #2003

Merged
merged 5 commits into from
Apr 24, 2024

Conversation

Demilivor
Copy link
Contributor

closes #1407

This PR adds automatic reconnection to the Postgres/MySQL database with 1-second interval if connection was not established instead of exit(1)

@Demilivor Demilivor marked this pull request as draft April 18, 2024 07:38
@Demilivor
Copy link
Contributor Author

Hello @an-tao

Could you please review this PR. Please feel free to suggest improvements.

@@ -436,12 +436,23 @@ DbConnectionPtr DbClientImpl::newConnection(trantor::EventLoop *loop)
}
// Reconnect after 1 second
auto loop = closeConnPtr->loop();
loop->runAfter(1, [weakPtr, loop, closeConnPtr] {
trantor::TimerId reconnect_timer_id = loop->runEvery(
1, [weakPtr, loop, closeConnPtr, reconnect_timer_id] {
Copy link
Member

@an-tao an-tao Apr 18, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect the reconnect_timer_id cannot be captured correctly. Consider using a smart pointer.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for suggestion, I will change this as soon as I finish testing.

I hope to finish today.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reworked the code to make it work in this scenario also:

  1. Database down
  2. Start drogon application
  3. drogon tries to create an initial connection to the database.
    4a) behavior of the previous version and the code without changes from this PR: unexpected exit.
    4b) second version: If the initial connection was not established, there will be a new attempt to connect to the database after 1 second.

@hwc0919
Copy link
Member

hwc0919 commented Apr 19, 2024

Thanks for the PR!
Could you put the init() methods right after the constructor? That really helps the code review. The diff viewer now displays in a stupid way...

@Demilivor
Copy link
Contributor Author

Thanks for the PR! Could you put the init() methods right after the constructor? That really helps the code review. The diff viewer now displays in a stupid way...

Done!

@Demilivor Demilivor changed the title [DRAFT] Implemented database reconnection loop Implemented database reconnection loop Apr 19, 2024
@Demilivor Demilivor marked this pull request as ready for review April 19, 2024 04:52
@Demilivor
Copy link
Contributor Author

My latest update failed test:

In test case PgPipelineTest
/home/admin/arenadata_drogon/drogon/orm_lib/tests/pipeline_test.cpp:37  FAILED:
  Reason: PgPipelineTest_init(0) what():SQL execution timeout

I just reproduced the issue on my personal PC. I'm going to investigate this failure on Monday.

@hwc0919
Copy link
Member

hwc0919 commented Apr 19, 2024

closes #1407

This PR adds automatic reconnection to the Postgres/MySQL database with 1-second interval if connection was not established instead of exit(1)

To be noted, auto-reconnect is already supported. This exit(-1) happens on some edge cases, such as reaching open socket limits.

@rvasin
Copy link

rvasin commented Apr 19, 2024

To be noted, auto-reconnect is already supported. This exit(-1) happens on some edge cases, such as reaching open socket limits.

From wiki

The connections managed by DbClient are always reconnected, so users don't need to care about the connection status. They are almost always connected.

The problem is not with "some edge cases" because simple drop of connection to PostgreSQL for few seconds makes the whole Drogon's application exit(-1).

@Demilivor
Copy link
Contributor Author

The issue with PgPipelineTest should be fixed now.

@Demilivor
Copy link
Contributor Author

One more update:

  1. Added connection initialization to make db_listener_test test pass.

I'm not familiar with the internal Drogon tests. Do you have a guide how I can prepare the test environment on my PC and run all Drogon tests before updating PR?

@an-tao
Copy link
Member

an-tao commented Apr 22, 2024

One more update:

  1. Added connection initialization to make db_listener_test test pass.

I'm not familiar with the internal Drogon tests. Do you have a guide how I can prepare the test environment on my PC and run all Drogon tests before updating PR?

You could refer to the github actions configuration: https://github.com/drogonframework/drogon/tree/master/.github/workflows

Copy link
Member

@hwc0919 hwc0919 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good

@an-tao an-tao merged commit e79d517 into drogonframework:master Apr 24, 2024
31 of 34 checks passed
This pull request was closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

After restarting postgres server manually a drogon project quits unexpectedly
4 participants