Incorporated faster line count

[llogiq]

Thanks for the hint; here's the faster linecount for your ripgrep utility.

[llogiq]

See newlinebench for more information.

[Byron]

I am so looking forward to see this PR land, just because line counts are enabled by default and are something people do expect to see anyway. Looking at the benchmarks on @BurntSushi 's blog, being faster at this will be absolutely noticeable !

[BurntSushi]

It will likely only be noticeable when searching very large files.

But yes, I believe @llogiq is working on putting this in a crate since there have been other developments. I would also like to understand the algorithm and audit it for safety before merging, so it might be a bit. Hang tight.

[Byron]

Sounds very reasonable ! I will happily use the fastest and best 'out-of-the' best experience then, to soon be able to tout it just became a little bit faster :).

[llogiq]

Yeah, I want to strip down the use of unsafe, put that in its own crate, document the bit twiddling and the rationale better and then I'll update.

Of course you can pull this now (as it'll already be faster), and I'll push a new PR when it's ready.

[BurntSushi]

I don't understand why you rewrote my Cargo.toml? Could you please not do that? (There are several stylistic choices that I don't like.)

[BurntSushi]

@llogiq Thanks! I have a nit about changes to Cargo.toml.

Other than that, I need to audit bytecount before I can merge this. I'm not sure when I'll get to that. Maybe this weekend? It would help if there was an explanation of how it works. cc @Veedrac

[Veedrac]

@BurntSushi I've got a new version that should be easier to audit.

The basic idea is that the function vector_not turns any null bytes in a usize into 1s and any others into 0s. I go along four usizes at a time, adding these into a counts array. Every 255 quadruplets, I flush the bytewise counts into the total count to prevent any byte from overflowing.

[llogiq]

I have a somewhat thorough explanation on my blog.

[llogiq]

Any updates on this?

[BurntSushi]

No.

On Oct 9, 2016 5:25 PM, "llogiq" [email protected] wrote:

Any updates on this?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#11 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAb34gzTxlpTX36PrUZNFnmCevI73D8aks5qyVvkgaJpZM4KEkOo
.

[Veedrac]

FWIW, a newer version has SIMD support, which is a lot faster.

[BurntSushi]

I merged this in 4368913 with a squash and a regression test for #128. Sorry it took so long @llogiq but thank you!

[BurntSushi]

This did wind up with quite a nice speed up! Comparing old master (rg-master) with rg using avx-accel.

First, a base line with no line counting:

[andrew@Cheetah subtitles] time rg-master 'Sherlock Holmes' OpenSubtitles2016.raw.en | wc -l
5107

real    0m1.578s
user    0m1.213s
sys     0m0.363s
[andrew@Cheetah subtitles] time rg 'Sherlock Holmes' OpenSubtitles2016.raw.en | wc -l
5107

real    0m1.533s
user    0m1.140s
sys     0m0.390s

And now with line counting:

[andrew@Cheetah subtitles] time rg-master -n 'Sherlock Holmes' OpenSubtitles2016.raw.en | wc -l
5107

real    0m4.260s
user    0m3.867s
sys     0m0.397s
[andrew@Cheetah subtitles] time rg -n 'Sherlock Holmes' OpenSubtitles2016.raw.en | wc -l
5107

real    0m2.131s
user    0m1.810s
sys     0m0.323s

Twice as fast!

[llogiq]

Cool!