Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use optimized implementation of bloom filter #4213

Open
ozgrakkurt opened this issue May 13, 2023 · 4 comments
Open

Use optimized implementation of bloom filter #4213

ozgrakkurt opened this issue May 13, 2023 · 4 comments
Labels
enhancement Any new improvement worthy of a entry in the changelog

Comments

@ozgrakkurt
Copy link

Hey!

I implemented https://github.com/ozgrakkurt/sbbf-rs.

It is an implementation of parquet bloom filters, I checked it against the implementation at parquet2 and it produces same output.

I would like to integrate it here if it makes sense to do.

A big problem with it may be that it has different implementations including a aarch64::neon which wouldn't get tested on GitHub CI.

@ozgrakkurt ozgrakkurt added the enhancement Any new improvement worthy of a entry in the changelog label May 13, 2023
@tustvold
Copy link
Contributor

Do you have any performance benchmark results you could share?

@ozgrakkurt
Copy link
Author

I added it to the benchmarks here: crepererum-oss/pdatastructs.rs#126.

@ozgrakkurt
Copy link
Author

ozgrakkurt commented Jun 20, 2023

@tustvold added benchmark against parquet2 implementation on the repo can you run it if you have time?

I ran it on an aarch64 cpu and these are the results I get:

INSERT
parquet2 -> 76 ns
sbbf-rs -> 3.5ns

CONTAINS (this seems to be dominated by hashing time and dynamic dispatch in sbbf-rs, also seems like the compiler optimizes the code in parquet2 pretty well)
parquet2 -> 3.4 ns
sbbf-rs -> 2.8 ns

CONTAINS (without hashing and dynamic-dispatch in sbbf-rs, requires modifying library a little specifically for aarch64 since all aarch64 cpus support `neon` SIMD instructions, don't want to release with this optimization since not sure if it is worth complicating the code)
parquet2 -> 1.9 ns
sbbf-rs -> 500 ps

couldn't bench the parquet implementation since the struct for bloom filter seemed non-trivial to construct

@ozgrakkurt
Copy link
Author

Added parquet impl by copy-pasting the bloom_filter file

results on mac m1:
image

results on VPS (x86_64) with 1 gb ram and 1vcore cpu:
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Any new improvement worthy of a entry in the changelog
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants