Add xeval

[nayeemrmn]

Experimenting with moving deno xeval here.
CC @bartlomieju @kevinkassimo

[kevinkassimo]

It is embarrassing, but I have to admit I just spot a bug with the chunks implementation I did before... The multi-char delimiter matching is throwing away incorrect parts, resulting in things like !DEDELIM!DELIM! incorrectly delimited with -d 'DELIM' set... We need KMP or Boyer Moore for this

[bartlomieju]

@kevinkassimo maybe it's worth fixing the problem in this PR? IMHO it makes sense to move xeval to deno_std

[kevinkassimo]

@bartlomieju I had a PR fixing the behavior in the deno repo. I’ll wait for it to be reviewed and see we can completely move it here

[kevinkassimo]

I was also thinking if we should move chunks to io/? Since it takes a Reader interface and might be also useful in other scenarios.

[bartlomieju]

I was also thinking if we should move chunks to io/? Since it takes a Reader interface and might be also useful in other scenarios.

Yes that makes sense, I'd also suggest adding lines function that is an alias for chunks - it's been requested many times.

[nayeemrmn]

Well it behaves like String::split() but for Readers. I would call it readAndSplit() or maybe even just split(). chunks() is vague.

[kevinkassimo]

@Narendra-Kamath I like the idea of including split in the name, but readAndSplit sounds a bit too long? Maybe readSplit() or readDelim() (align with readLine)? Also we can add a maxSplit property to it, such that readSplit(reader, "\n", { maxSplit: 2 }) will yield at most 2 strings.

Also we might want to distinguish async function* readSplit(r: Reader, delim: Uint8Array): AsyncIterableIterator<Uint8Array> and readStrSplit(r: Reader, delim: string): AsyncIterableIterator<string>. (This change should be quite simple)

Also I noticed https://github.com/denoland/deno_std/blob/master/io/readers.ts . We can maybe also implement a DelimitedReader

[nayeemrmn]

distinguish async function* readSplit(r: Reader, delim: Uint8Array): AsyncIterableIterator<Uint8Array> and readStrSplit(r: Reader, delim: string): AsyncIterableIterator<string>.

Yeah. Most accurately it reads, decodes to string and splits. readStrSplit() checks every box.

[bartlomieju]

Also I noticed https://github.com/denoland/deno_std/blob/master/io/readers.ts . We can maybe also implement a DelimitedReader

👍 additionally Rust has lines method implemented on BufReader - I believe we should replicate this function on Deno's BufReader

As for the renaming of chunks I'd suggest readDelimitedChunks.

@nayeemrmn can you update the PR? There are a few issues waiting on this functionality :)

[nayeemrmn]

@bartlomieju I think moving chunks() should be a separate PR. It would get merged a lot sooner and I can rebase this one. @kevinkassimo?

[kevinkassimo]

I'm fine with landing this first (as long as we don't expose chunks or equivalent here)

[bartlomieju]

@ry please review

[ry]

LGTM