LanguageClient failing to initialize language server on 1.62.1

[bcanzanella]

We're running into an issue that seems to have been introduced with 1.62.1.

Upon initialization of our LanguageClient in our VSCode extension, it's immediately triggering a closed notification on LanguageClient.clientOptions.errorHandler.closed (here)

Insiders 1.63.0 also has the problem. It happens while debugging as well as when installing from the marketplace.

If we downgrade to 1.61.2 or 1.60.2 there are no problems.

• We checked the lsp-sample using 1.62.1 and using the same client + server libraries and there's no error.
• We changed our retry logic to retry indefinitely, and at some point it will complete. it seems like it's related to the client.onReady() promise never resolving [?]
• What else can we try, this is essentially affecting all our users.

Does this issue occur when all extensions are disabled?: N/A, this is inside a VSCode extension

Version: 1.62.1
Commit: f4af3cb
Date: 2021-11-05T09:23:14.144Z
Electron: 13.5.2
Chrome: 91.0.4472.164
Node.js: 14.16.0
V8: 9.1.269.39-electron.0
OS: Darwin x64 20.6.0

Happens in both VSC 1.62.2 and VSCInsiders 1.63.0
Steps to Reproduce:

1. install CodeStream from the marketplace
2. close VSCode, then reopen it. 75% of the time you'll see a toast notification error

[ycrepeau]

Moved back from 1.62 to 1.61.2 and the problem is gone. So, you hypothesis is right, at least on Linux (Ubuntu) machines.

Electron 13.5.1 vs 13.5.2

.

[chrmarti]

Sorry about the breakage, this is from a fix to address an elevation of privilege vulnerability #136771 (CVE-2021-42322). Starting with 1.62.1 you need to pass the command line argument --ms-enable-electron-run-as-node to the VSCode binary in addition to ELECTRON_RUN_AS_NODE=1 to use it as Node.js.

[bcanzanella]

@chrmarti could you clarify your solution for me? How / where are those additional arguments set?
We are a VSCode extension and have no ability to change how the user launched VSCode.

[jonathanm-tkf]

@chrmarti workaround for mac? thanks

[alexdima]

@bcanzanella It is not clear to me that this is a regression caused by our security fix, because our security fix changes Electron's behavior in 100% of cases, not just in 75% of cases.

Can you please clarify if this reproduces in 1.62.0 or not? Is 1.62.1 the first bad version?

[bcanzanella]

So the 75%/25% seems like an difference between running VSCode on a "read-only" volume, in my case from the Downloads folder rather than installed directly. in this case the error happens immediately 75% of the time (via "read-only" volume), and the other 25% seems to load, but later the LSP agent restarts multiple times and makes CodeStream unusable, but let's shelve that for the moment...

100% of the time the following is true:

1.62.0 WORKS
1.62.1 BAD
1.62.2 BAD

[alexdima]

@bcanzanella I have investigated and it looks like the forked process dist/agent.js exits with a SIGSEGV.

@deepak1556 Is there anything we can do to find out why this process crashes? AFAICT it is not loading any native node modules.

[atscott]

@bcanzanella I have investigated and it looks like the forked process dist/agent.js exits with a SIGSEGV.

@deepak1556 Is there anything we can do to find out why this process crashes? AFAICT it is not loading any native node modules.

This is exactly the issue we are experiencing in the Angular Language Service when trying to run ngcc as a forked process. In 1.61.1, 1.61.2, and the insider version, the process fails with SIGEGV and a null code. However, this works on 1.61.0.

[deepak1556]

@bcanzanella when I repro the crash with codestream extension, the stacks indicate crash happens during compilation of the dist/agent.js file, the following is the translated callsite

"%cCHILD_PROCESS 66072: spawn {
%c  execArgv: [],
%c  cwd: '/Users/demohan/github/electron-quick-start',
%c  silent: true,
%c  stdio: [ 'pipe', 'pipe', 'pipe', 'ipc' ],
%c  env: { ELECTRON_RUN_AS_NODE: 1 },
%c  execPath: '/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)',
%c  shell: false,
%c  args: [
%c    '/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)',
%c    '--ms-enable-electron-run-as-node',
%c    '/Users/demohan/.vscode/extensions/codestream.codestream-12.1.0/dist/agent.js',
%c    '--node-ipc',
%c    '--clientProcessId=66072'
%c  ],
%c  detached: false,
%c  envPairs: [....],
%c  file: '/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)',
%c  windowsHide: false,
%c  windowsVerbatimArguments: false
%c}

When I compare it with https://github.com/TeamCodeStream/codestream/blob/7c7d4d98f9d93d0ee70690d3c12093995823e60e/atom/lib/workspace/agent/connection.ts#L419-L423 the args don't match, can you share the code calling spawn ?

I was trying to play with the --no-lazy flag and the crash no longer repros, not sure why it makes a difference.

// crashes
ELECTRON_RUN_AS_NODE=1 /Applications/Visual\ Studio\ Code.app/Contents/Frameworks/Code\ Helper\ \(Renderer\).app/Contents/MacOS/Code\ Helper\ \(Renderer\) /Users/demohan/.vscode/extensions/codestream.codestream-12.1.0/dist/agent.js --node-ipc --ms-enable-electron-run-as-node

// does not crash
ELECTRON_RUN_AS_NODE=1 /Applications/Visual\ Studio\ Code.app/Contents/Frameworks/Code\ Helper\ \(Renderer\).app/Contents/MacOS/Code\ Helper\ \(Renderer\) /Users/demohan/.vscode/extensions/codestream.codestream-12.1.0/dist/agent.js --node-ipc --ms-enable-electron-run-as-node --no-lazy

[deepak1556]

@atscott can you share some minimal steps to repro the issue, thanks!

[bcanzanella]

@deepak1556 here is the code that spawns it: https://github.com/TeamCodeStream/codestream/blob/develop/vscode/src/agent/agentConnection.ts#L257

[bcanzanella]

also, @deepak1556 does the order of those arguments matter? in your debugging code --ms-enable-electron-run-as-node is before the script arg (agent.js)...

args: [
%c    '/Applications/Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)',
%c    '--ms-enable-electron-run-as-node',
%c    '/Users/demohan/.vscode/extensions/codestream.codestream-12.1.0/dist/agent.js',
%c    '--node-ipc',
%c    '--clientProcessId=66072'
%c  ],

...but in your manual testing the --ms-enable-electron-run-as-node arg is after agent.js

[deepak1556]

Order doesn't matter --ms-enable-electron-run-as-node argument is parsed and removed early on in the process startup.

[atscott]

Hi @deepak1556, sorry if these aren't minimal enough directions! Let me know what I can do to provide something better:

1. Clone our extension repo
2. Open the repository with vscode 1.62.1 or above
3. Install dependencies with yarn && cd integration/project/ && yarn
4. Run the build script with ./scripts/build.sh
5. For debugging, put a breakpoint on the line that calls fork and one on the close listener
6. Launch the extension development host (Dev Client + Server run configuration)
7. In the extension host, open the integration project and then open the app.component.html template file
   -- debugger should stop on the fork and if you continue, you will see that the debugger stops in the childProcess.on('close') with a null code.
   You can have the fork retry by running the Angular: Restart Angular Language server command in the extension host.

[atscott]

I also attempted to add the --ms-enable-electron-run-as-node flag to the child_process fork but that didn't work.

Edit: The arguments that are being passed to spawn from the fork call are:
0 (command - I'm running vscode from the downloads folder): '/.../AppTranslocation/.../Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)'
1 (args): (5) ['/Users/atscott/github/vscode-ng-language-service/…/@angular/compiler-cli/bundles/ngcc/main-ngcc.js', '--tsconfig', '/Users/atscott/github/vscode-ng-language-service/integration/project/tsconfig.json', '--typings-only', '--ms-enable-electron-run-as-node']
2 (options):

cwd:'/Users/atscott/github/vscode-ng-language-service/integration/project'
env:{ELECTRON_RUN_AS_NODE: 1}
execArgv:(0) []
execPath:'/.../AppTranslocation/.../Visual Studio Code.app/Contents/Frameworks/Code Helper (Renderer).app/Contents/MacOS/Code Helper (Renderer)'
shell:false
silent:true
stdio:(4) ['pipe', 'pipe', 'pipe', 'ipc']

[deepak1556]

End-to-end debugging of the issue narrowed down the problem to a UAF scenario in the argument parser. I am getting a new electron build ready with the fix today. I will update here once the fix is rolled into our insiders for some testing. Sorry about the breakage!

[deepak1556]

For anyone available for early testing, here are some test builds. Tomorrow's insider should also contain the fix.

macOS x64
windows x64
linux x64

[bcanzanella]

thanks @deepak1556 we have confirmation from a few CodeStream developers that it is fixed in the macOS version

[atscott]

@deepak1556 I can also confirm that this resolves the issue with forking a process for ngcc in the Angular language service extension. Thank you!

[deepak1556]

Today's insider Version: 1.63.0-insider 6a25ae3 , carries the fix. I will close this optimistically based on the testing. The fix will also be made available in recovery release 1.62.3 later this week.