Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MDEV-32777 DDL operations very slow under macOS #3136

Open
wants to merge 1 commit into
base: 10.5
Choose a base branch
from

Conversation

DaveGosselin-MariaDB
Copy link
Member

@DaveGosselin-MariaDB DaveGosselin-MariaDB commented Mar 18, 2024

macOS is a development platform for our product but calls to fcntl(F_FULLFSYNC) are too slow for developer needs. macOS offers at least three different disk flush calls, making tradeoffs between reliability and performance (see analysis notes on this MDEV). On macOS only and as a compromise, we will back out the change to call fcntl(F_FULLFSYNC), sacrificing a full flush in favor of more performant writes.

@CLAassistant
Copy link

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@DaveGosselin-MariaDB DaveGosselin-MariaDB force-pushed the 11.4-mdev-32777 branch 2 times, most recently from c99a028 to c5bbace Compare March 18, 2024 18:43
@anson1014
Copy link
Contributor

anson1014 commented Mar 19, 2024

In the description of MDEV-32777, I can see very clearly the times listed to run the attached DDL script in macOS and how it is much slower than other platforms.

Just a suggestion, but perhaps you can run the same script with your changes and then list the time taken to further your case for how this change will be more performant?

@DaveGosselin-MariaDB
Copy link
Member Author

Hi @anson1014 there is a comment on the ticket with results but it's limited to Developers. The relevant part you ask about is:

For the attached DDL SQL script, and the following configurations, see the below performance figures (git sha 70de4075a18c12ed7e55db9a49e66eb6cff308b2).
innodb_flush_log_at_trx_commit=1, F_FULLFSYNC: ~5.0s
innodb_flush_log_at_trx_commit=0, F_FULLFSYNC: ~4.7s
innodb_flush_log_at_trx_commit=1, F_BARRIERFSYNC: ~2.2s
innodb_flush_log_at_trx_commit=0, F_BARRIERFSYNC: ~2.1s
innodb_flush_log_at_trx_commit=1, fsync: ~300ms
innodb_flush_log_at_trx_commit=0, fsync: ~300ms

@anson1014
Copy link
Contributor

Thanks for that, I see now where you mentioned the results on the Jira. Looks good!

Copy link
Contributor

@anson1014 anson1014 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance gains are apparent, however I cannot speak to the contextual impact of these changes.

@vuvova
Copy link
Member

vuvova commented Mar 29, 2024

why couldn't you simply set innodb_flush_log_at_trx_commit=0 ? That would be as fast or may be even faster, wouldn't it?

@DaveGosselin-MariaDB
Copy link
Member Author

why couldn't you simply set innodb_flush_log_at_trx_commit=0 ? That would be as fast or may be even faster, wouldn't it?

I attempted that as well. Please see my data and results in comment (restricted to Developers): https://jira.mariadb.org/browse/MDEV-32777?focusedCommentId=277114&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-277114

macOS is a development platform for our product but calls to
fcntl(F_FULLFSYNC) are too slow for developer needs.  macOS offers
at least three different disk flush calls, making tradeoffs between
reliability and performance (see analysis notes on this MDEV).
On macOS only and as a compromise, we will back out the change to
call fcntl(F_FULLFSYNC), sacrificing a full flush in favor of more
performant writes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
4 participants