[MDEV-34009] Introduce server-initiated instant failover mechanism #3231
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This adds the system variables `INSTANT_FAILOVER_TARGET` and
`INSTANT_FAILOVER_MODE` (`OFF`/`ON`/`ALL`), and the error code
`ER_INSTANT_FAILOVER`.
When `INSTANT_FAILOVER_MODE=ON`, the server will immediately respond
to newly-connected clients with an error packet containing the error code
4196 (`ER_INSTANT_FAILOVER`) and a specially-formatted message:
|Arbitary human-readable message|value of INSTANT_FAILOVER_TARGET
For example:
|Server is directing clients to the alternative server 'other-mariadb-server.company.com:3307'.|other-mariadb-server.company.com:3307
Updated and compatible clients can parse this message and redirect
appropriately, or display the human-readable message if they do not wish to
follow this redirection. Older clients will display the message in its
entirety, and end users should at least have an idea of what's going on.
In my earliest implementation, the sending of the `ER_INSTANT_FAILOVER`
error packet by the MariaDB server depended on the exploitation of the
client vulnerability https://jira.mariadb.org/browse/CONC-648 (“Client
improperly accepts error packets prior to TLS handshake”), which I
discovered during its implementation.
The server should obviously be able to redirect clients which don't suffer
from this severe vulnerability.
In order to do this, we need to move the redirection handling into
`native_password_authentication()`. This awkward arrangement is
necessitated by the total entanglement of the APPLICATION-layer
authentication code (e.g. username+password for "native" authentication)
with the TRANSPORT-layer security mechanism (TLS literally stands for
Transport Layer Security).
An unfortunate consequence of this is that the redirection mechanism will
not work unless the client is using the "native" authentication plugin, or
until other authentication plugins are similarly updated similarly.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
MariaDB servers can optionally listen for client connections on an additional TCP port (configured with the system variable `EXTRA_PORT`) in addition to the "normal" TCP port (default 3306). See https://mariadb.com/kb/en/thread-pool-in-mariadb/#configuring-the-extra-port for more information. This `EXTRA_PORT` is intended for emergency and/or administrative use, and we should therefore include it from redirection in `INSTANT_FAILOVER_MODE=ON`, just as we do for non-network based connections (Unix sockets and named pipes). The code required to check for connections to the EXTRA_PORT is greatly complicated by the fact that the `thd->net->vio->mysql_socket` and `thd->net->vio->local` data structures are not reliably or fully populated at the point where they are passed into `parse_client_handshake_packet. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
|
|
4 tasks
dlenski
changed the title
The updated libmariadb parses the `ER_INSTANT_FAILOVER` error packets (with the message component formatted as `|Human-readable message|value of INSTANT_FAILOVER_TARGET system variable`) and reconnects accordingly. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
This option is ENABLED by default. The `main.mysqld--help` MTR test is updated as well, to reflect the addition of this option to the `mysqld --help` output. All new code of the whole pull request, including one or several files that are either new files or modified ones, are contributed under the BSD-new license. I am contributing on behalf of my employer Amazon Web Services, Inc.
This MTR test verifies the basic functionality of the
INSTANT_FAILOVER_{MODE,TARGET} system variables, including:
1. `INSTANT_FAILOVER_MODE=ON` as well as `ALL`
2. The handling of the extra port and local-socket connections (redirected
in mode `ALL`, but not in mode `ON`)
3. The behavior of the client both with `--follow-instant-failovers` (the
default) and with `--disable-follow-instant-failovers`
4. The client's handling of redirect loops (>=8 redirections causes
`ER_INSTANT_FAILOVER`)
5. The ability to turn `INSTANT_FAILOVER_MODE` back to `OFF`, and to
resume connections without redirection.
All new code of the whole pull request, including one or several files
that are either new files or modified ones, are contributed under the
BSD-new license. I am contributing on behalf of my employer Amazon Web
Services, Inc.
dlenski
force-pushed
the
MDEV-34009_server_initiated_instant_failover
branch
from
b72eba6
to
944ad78
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a pared-down version of #3224, which only introduces the server-initiated instant failover mechanism and does not address any of the pre-existing TLS issues which were discovered or highlighted during its development.
Upstream developers chose a different approach (see #1472 and mariadb-corporation/mariadb-connector-c#137) for MDEV-15935 ("connection redirection").
I maintain that a simple and obligatory server-directed failover mechanism will be a very useful feature for many users, so I'm reintroducing it here under the alternative name of "instant failover."
Description
This PR proposes a new feature in the MariaDB client-server protocol: a server-initiated failover which can redirect new client connections to an alternate endpoint at the application layer.
It also provides an initial server-side implementation of that feature; the corresponding client-side implementation is in mariadb-corporation/mariadb-connector-c#246.
We want the MariaDB server to be able to tell clients connecting to it, “Sorry, this server is unavailable. Connect to an alternate server instead.” This mechanism is inspired by HTTP 302 (“temporary redirect”) mechanism familiar to all developers of web applications, and is intended to have similar semantics and security properties, since these have now been widely deployed and tested for decades.
How can this PR be tested?
Automated testing
main.instant_failoverMTR test functionally verifies the instant failover mechanism.Manual testing
sql/mariadbd --instant-failover-mode=ON --instant-failover-target='some-other-mariadb.server.com:3306' --port=3306 --extra-port=3307 --socket=/tmp/mysql.sockEXTRA_PORTand to the Unix socket are not redirected:client/mariadb --host 127.0.0.1 --port=3306 -uUSERNAME -pPASSWORD→ redirected toINSTANT_FAILOVER_TARGET-client/mariadb --host 127.0.0.1 --port=3307 -uUSERNAME -pPASSWORD→ not redirected (EXTRA PORT)client/mariadb --socket=/tmp/mysql.sock -uUSERNAME -pPASSWORD→ not redirected (local/non-network-based connection)Basing the PR against the correct MariaDB version
PR quality check