Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(lsp): Add benchmark for performance on a large real-world repo #23395

Merged
merged 7 commits into from
Apr 16, 2024

Conversation

nathanwhit
Copy link
Member

@nathanwhit nathanwhit commented Apr 16, 2024

This PR adds a benchmark intended to measure how the LSP handles larger repos, as well as its performance on a more realistic workload.

The repo being benchmarked is deco-cx/apps which has been vendored along with its dependencies. It's included as a git submodule as its fairly large. The LSP requests used in the benchmark are the actual requests sent by VSCode as I opened, modified, and navigated around a file (to simulate an actual user interaction).

The main motivation is to have a more realistic benchmark that measures how we do with a large number of files and dependencies. The improvements made from 1.42 to 1.42.3 mostly improved performance with larger repos, so none of our existing benchmarks showed an improvement.

Here are the results for the changes made from 1.42 to 1.42.3 (the new benchmark is the last one listed):

1.42.0

Starting Deno benchmark
-> Start benchmarking lsp
   - Simple Startup/Shutdown 
      (10 runs, mean: 379ms)
   - Big Document/Several Edits 
      (5 runs, mean: 1142ms)
   - Find/Replace
      (10 runs, mean: 51ms)
   - Code Lens
      (10 runs, mean: 443ms)
   - deco-cx/apps Multiple Edits + Navigation
      (5 runs, mean: 25121ms)
<- End benchmarking lsp

1.42.3

Starting Deno benchmark
-> Start benchmarking lsp
   - Simple Startup/Shutdown 
      (10 runs, mean: 383ms)
   - Big Document/Several Edits 
      (5 runs, mean: 1135ms)
   - Find/Replace
      (10 runs, mean: 55ms)
   - Code Lens
      (10 runs, mean: 440ms)
   - deco-cx/apps Multiple Edits + Navigation
      (5 runs, mean: 11675ms)
<- End benchmarking lsp

@nathanwhit nathanwhit merged commit 422cff1 into denoland:main Apr 16, 2024
17 checks passed
@nathanwhit nathanwhit deleted the lsp-apps-bench branch April 16, 2024 19:26
nathanwhit added a commit that referenced this pull request Apr 16, 2024
Forgot to add the step to clone the new submodule added in #23395 😅.
littledivy pushed a commit to littledivy/deno that referenced this pull request Apr 19, 2024
…enoland#23395)

This PR adds a benchmark intended to measure how the LSP handles larger
repos, as well as its performance on a more realistic workload.

The repo being benchmarked is
[deco-cx/apps](https://github.com/deco-cx/apps) which has been vendored
along with its dependencies. It's included as a git submodule as its
fairly large. The LSP requests used in the benchmark are the actual
requests sent by VSCode as I opened, modified, and navigated around a
file (to simulate an actual user interaction).

The main motivation is to have a more realistic benchmark that measures
how we do with a large number of files and dependencies. The
improvements made from 1.42 to 1.42.3 mostly improved performance with
larger repos, so none of our existing benchmarks showed an improvement.

Here are the results for the changes made from 1.42 to 1.42.3 (the new
benchmark is the last one listed):

**1.42.0**

```test
Starting Deno benchmark
-> Start benchmarking lsp
   - Simple Startup/Shutdown 
      (10 runs, mean: 379ms)
   - Big Document/Several Edits 
      (5 runs, mean: 1142ms)
   - Find/Replace
      (10 runs, mean: 51ms)
   - Code Lens
      (10 runs, mean: 443ms)
   - deco-cx/apps Multiple Edits + Navigation
      (5 runs, mean: 25121ms)
<- End benchmarking lsp
```

**1.42.3**

```text
Starting Deno benchmark
-> Start benchmarking lsp
   - Simple Startup/Shutdown 
      (10 runs, mean: 383ms)
   - Big Document/Several Edits 
      (5 runs, mean: 1135ms)
   - Find/Replace
      (10 runs, mean: 55ms)
   - Code Lens
      (10 runs, mean: 440ms)
   - deco-cx/apps Multiple Edits + Navigation
      (5 runs, mean: 11675ms)
<- End benchmarking lsp
```
littledivy pushed a commit to littledivy/deno that referenced this pull request Apr 19, 2024
Forgot to add the step to clone the new submodule added in denoland#23395 😅.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants