-
Notifications
You must be signed in to change notification settings - Fork 106
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Faster UTF8 validation #705
Comments
It seems to use vectorized (SIMD) instructions, i would say it goes a bit to far to have this kind of optimization. The UTF8 validation overhead is only the tiniest percentage of the whole workload. Not sure if optimization of this would give you any noticable performance gain. I wonder why boost locale does not have a validation function and select an optimized version based on CPU architecture. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Just making the project aware of this faster algorithm. https://lemire.me/blog/2020/10/20/ridiculously-fast-unicode-utf-8-validation/
Possible ways to take advantage of this are to provide some kind of hook for user code to provide it's own UTF8 validation, or a compile time option to specify a UTF8 validation function as a dependency.
The text was updated successfully, but these errors were encountered: