Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

s2: Add full Snappy output support #396

Merged
merged 4 commits into from
Jun 11, 2021
Merged

s2: Add full Snappy output support #396

merged 4 commits into from
Jun 11, 2021

Conversation

klauspost
Copy link
Owner

@klauspost klauspost commented Jun 9, 2021

S2 can now fully replace snappy and generate snappy output at all compression levels, including streaming.

Comparison of webdevdata.org-2015-01-07-subset,
53927 files, total input size: 4,014,735,833 bytes. amd64, single goroutine used:

Encoder Size MB/s
snappy.Encode 1128706759 725.59
s2.EncodeSnappy 1093823291 899.16
s2.EncodeSnappyBetter 1001158548 578.49
s2.EncodeSnappyBest 944507998 66.00

Comparison of different streams, AMD Ryzen 3950x, 16 cores. Size and throughput:

File snappy.NewWriter S2 Snappy S2 Snappy, Better S2 Snappy, Best
nyc-taxi-data-10M.csv 1316042016 - 517.54MB/s 1307003093 - 8406.29MB/s 1174534014 - 4984.35MB/s 1115904679 - 177.81MB/s
enwik10 5088294643 - 433.45MB/s 5175840939 - 8454.52MB/s 4560784526 - 4403.10MB/s 4340299103 - 159.71MB/s
10gb.tar 6056946612 - 703.25MB/s 6208571995 - 9035.75MB/s 5741646126 - 2402.08MB/s 5548973895 - 171.17MB/s
github-june-2days-2019.json 1525176492 - 908.11MB/s 1476519054 - 12625.93MB/s 1400547532 - 6163.61MB/s 1321887137 - 200.71MB/s
consensus.db.10gb 5412897703 - 1054.38MB/s 5354073487 - 12634.82MB/s 5335069899 - 2472.23MB/s 5201000954 - 166.32MB/s

S2 can now fully replace snappy and generate snappy output at all compression levels, including streaming.
# Conflicts:
#	s2/encodeblock_amd64.s
@klauspost klauspost merged commit 5453ad3 into master Jun 11, 2021
@klauspost klauspost deleted the s2-full-snappy-compat branch June 11, 2021 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant