Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

improve performance of TOML parsing by reading from memory instead of from file #1751

Merged
merged 6 commits into from
Apr 7, 2020

Conversation

KristofferC
Copy link
Sponsor Member

@KristofferC KristofferC commented Apr 6, 2020

Reading a bunch of small things from a file tends to be much slower than just reading the whole file in one chunk and operating on memory (especially with Julia now locking the stream for every read). None of the files we parse are big enough that operating on a file makes sense.

Benchmark:

function parse_registry()
    path = joinpath(homedir(), ".julia/registries/General")
    for (root, dirs, files) in walkdir(path)
       for file in files
           if endswith(file, ".toml")
               TOML.parsefile(joinpath(root, file))
           end
       end
   end
end

Before:

696.417 ms (2111063 allocations: 157.45 MiB)

After:

355.687 ms (2164815 allocations: 166.51 MiB)

@KristofferC
Copy link
Sponsor Member Author

Ah, this caught a too loose try catch.

@KristofferC KristofferC merged commit 47784de into master Apr 7, 2020
@KristofferC KristofferC deleted the kc/parse_speed branch April 7, 2020 07:10
KristofferC added a commit that referenced this pull request May 10, 2020
… from file (#1751)

* improve performance of TOML parsing by reading from memory instead of file

(cherry picked from commit 47784de)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants