Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[MDEV-34009] Introduce server-initiated instant failover mechanism #3231

Open
wants to merge 5 commits into
base: 11.5
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Prev Previous commit
Add main.instant_failover MTR
This MTR test verifies the basic functionality of the
INSTANT_FAILOVER_{MODE,TARGET} system variables, including:

1. `INSTANT_FAILOVER_MODE=ON` as well as `ALL`
2. The handling of the extra port and local-socket connections (redirected
   in mode `ALL`, but not in mode `ON`)
3. The behavior of the client both with `--follow-instant-failovers` (the
   default) and with `--disable-follow-instant-failovers`
4. The client's handling of redirect loops (>=8 redirections causes
   `ER_INSTANT_FAILOVER`)
5. The ability to turn `INSTANT_FAILOVER_MODE` back to `OFF`, and to
   resume connections without redirection.

All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
  • Loading branch information
dlenski committed Apr 29, 2024
commit 944ad7879a57b5eb6e4bd883dfc8eb1fe211945a
7 changes: 7 additions & 0 deletions mysql-test/main/instant_failover.cnf
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
!include include/default_my.cnf

[mysqld.1]
[email protected]_EXTRA_PORT

[ENV]
MASTER_EXTRA_PORT= @OPT.port
40 changes: 40 additions & 0 deletions mysql-test/main/instant_failover.result
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
call mtr.add_suppression("\\[Warning\\] Redirecting connection \\d+ via \\S+ to INSTANT_FAILOVER_TARGET=\\S+ \\(INSTANT_FAILOVER_MODE=\\w+\\)");
Verify that a TCP connection works with --disable-follow-instant-failovers (before enabling instant failover)
default_port_tcp_works
1
Verify that connections via local socket and extra port SUCCEED
connection default;
connect local_sock,localhost,root,,test;
disconnect local_sock;
connect tcp_sock,127.0.0.1,root,,test,$MASTER_EXTRA_PORT;
disconnect tcp_sock;
Enable instant failover in its default mode (ON)
connection default;
set global instant_failover_mode=ON;
set global instant_failover_target="127.0.0.1:$MASTER_EXTRA_PORT"
With --disable-follow-instant-failovers, connecting to the default port should now fail
With --follow-instant-failovers (the client library default), this should redirect to the extra port
connect OKAY,127.0.0.1,root,,test,$MASTER_MYPORT;
disconnect OKAY;
Setup a redirect loop, and verify that connections fail due to the loop
connection default;
set global instant_failover_target="127.0.0.1:$MASTER_MYPORT"
connect(127.0.0.1,root,,test,MASTER_MYPORT,MASTER_MYSOCK);
connect fail_con,127.0.0.1,root,,test,$MASTER_MYPORT;
ERROR HY000: Too many instant failovers (>= 8)
Change instant failover mode to ALL, and verify that even connections via local socket and extra port now FAIL due to the loop
connection default;
set global instant_failover_mode=ALL;
connect(localhost,root,,test,MASTER_MYPORT,MASTER_MYSOCK);
connect fail_con,localhost,root,,test;
ERROR HY000: Too many instant failovers (>= 8)
connect(127.0.0.1,root,,test,MASTER_EXTRA_PORT,MASTER_MYSOCK);
connect fail_con,127.0.0.1,root,,test,$MASTER_EXTRA_PORT;
ERROR HY000: Too many instant failovers (>= 8)
Turn instant failover back off
connection default;
set global instant_failover_mode=OFF;
set global instant_failover_target=DEFAULT;
Connections should now succeed again, even with --disable-follow-instant-failovers
default_port_tcp_works
1
64 changes: 64 additions & 0 deletions mysql-test/main/instant_failover.test
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
# We need to ignore the redirection warnings in the server logs, e.g.
# "Redirecting connection N via SSL/TLS to INSTANT_FAILOVER_TARGET=127.0.0.1:$MASTER_EXTRA_PORT (INSTANT_FAILOVER_MODE=ON)"
call mtr.add_suppression("\\[Warning\\] Redirecting connection \\d+ via \\S+ to INSTANT_FAILOVER_TARGET=\\S+ \\(INSTANT_FAILOVER_MODE=\\w+\\)");

##########
--echo Verify that a TCP connection works with --disable-follow-instant-failovers (before enabling instant failover)
--exec $MYSQL --disable-follow-instant-failovers --host=127.0.0.1 --port=$MASTER_MYPORT test -e "select 1 as default_port_tcp_works"

--echo Verify that connections via local socket and extra port SUCCEED
--connection default
--connect local_sock,localhost,root,,test
--disconnect local_sock
--connect tcp_sock,127.0.0.1,root,,test,$MASTER_EXTRA_PORT
--disconnect tcp_sock

##########
--echo Enable instant failover in its default mode (ON)
--connection default
set global instant_failover_mode=ON;
--echo set global instant_failover_target="127.0.0.1:\$MASTER_EXTRA_PORT"
--disable_query_log
--eval set global instant_failover_target="127.0.0.1:$MASTER_EXTRA_PORT"
--enable_query_log

##########
--echo With --disable-follow-instant-failovers, connecting to the default port should now fail
--error 1
--exec $MYSQL --disable-follow-instant-failovers --host=127.0.0.1 --port=$MASTER_MYPORT test -e "select 1"

--echo With --follow-instant-failovers (the client library default), this should redirect to the extra port
--connect OKAY,127.0.0.1,root,,test,$MASTER_MYPORT
--disconnect OKAY

##########
--echo Setup a redirect loop, and verify that connections fail due to the loop
--connection default
--echo set global instant_failover_target="127.0.0.1:\$MASTER_MYPORT"
--disable_query_log
--eval set global instant_failover_target="127.0.0.1:$MASTER_MYPORT"
--enable_query_log
--replace_result $MASTER_MYSOCK MASTER_MYSOCK $MASTER_MYPORT MASTER_MYPORT
--error ER_INSTANT_FAILOVER
--connect fail_con,127.0.0.1,root,,test,$MASTER_MYPORT

##########
--echo Change instant failover mode to ALL, and verify that even connections via local socket and extra port now FAIL due to the loop
--connection default
set global instant_failover_mode=ALL;
--replace_result $MASTER_MYSOCK MASTER_MYSOCK $MASTER_MYPORT MASTER_MYPORT
--error ER_INSTANT_FAILOVER
--connect fail_con,localhost,root,,test
--replace_result $MASTER_MYSOCK MASTER_MYSOCK $MASTER_EXTRA_PORT MASTER_EXTRA_PORT
--error ER_INSTANT_FAILOVER
--connect fail_con,127.0.0.1,root,,test,$MASTER_EXTRA_PORT

##########
--echo Turn instant failover back off
--connection default
set global instant_failover_mode=OFF;
set global instant_failover_target=DEFAULT;

--echo Connections should now succeed again, even with --disable-follow-instant-failovers
--error 0
--exec $MYSQL --disable-follow-instant-failovers --host=127.0.0.1 --port=$MASTER_MYPORT test -e "select 1 as default_port_tcp_works"