Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SSH disconnects when the laptop goes to sleep even for a short time while the native terminal is running #5755

Open
thechargedneutron opened this issue Oct 22, 2021 · 20 comments
Assignees
Labels
feature-request Request for new features or functionality ssh Issue in vscode-remote SSH
Milestone

Comments

@thechargedneutron
Copy link

  • VSCode Version: 1.62.0-insider
  • Local OS Version: macOS Big Sur 11.5.2
  • Remote OS Version: Ubuntu 18.04 LTS
  • Remote Extension/Connection Type: SSH
  • Logs:

Steps to Reproduce:

  1. Open native terminal in macOS and connect to a remote server. Connect to the same server with vscode.
  2. Close the laptop lid and instantly open it up. Alternatively, let the laptop sleep due to inactivity and then instantly wake it up using the trackpad.

The vscode is now disconnected and asks me to reload window. The macOS terminal is still connected to the remote and works perfectly. Is there a way to delay the session time-out time in vscode? It is a bit annoying to reload every time the screen goes off for a minute. I have to increase the sleep duration to tackle this issue.

Does this issue occur when you try this locally?: Yes
Does this issue occur when you try this locally and all extensions are disabled?: Yes

@github-actions github-actions bot added the ssh Issue in vscode-remote SSH label Oct 22, 2021
@tanhakabir
Copy link

@roblourens I'm guessing this has to do with the ssh process closing when the computer goes to sleep and when we try to reconnect, we aren't able to within the 5 times so we close the connection?

@roblourens
Copy link
Member

It should be trying to reconnect at this point. Please share the log from the Remote-SSH output channel, it should tell us why it failed to reconnect

@roblourens roblourens added the info-needed Issue requires more information from poster label Oct 29, 2021
@thechargedneutron
Copy link
Author

thechargedneutron commented Oct 30, 2021

@roblourens After I put the laptop lid down and instantly put it up, the password prompt comes, if I do not enter the password and click exit I get the following log. Though, my concern is that vscode should not ask for password again if I instantly open up my lid again (similar to macOS native terminal).

[19:57:04.914] [Forwarding server 52217] Got connection 3
[19:57:04.925] ------




[19:57:04.925] SSH Resolver called for "ssh-remote+`hostname`", attempt 2, (Reconnection)
[19:57:04.925] SSH Resolver called for host: `hostname`
[19:57:04.925] Setting up SSH remote `"hostname"`
[19:57:04.927] Acquiring local install lock: /var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-remote-ssh-76b40f39-install.lock
[19:57:04.934] Looking for existing server data file at /Users/thechargedneutron/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-76b40f39-34eeebdd66033633ac22efd5fab928ae37916168-0.65.8/data.json
[19:57:04.938] Found local server running: {"remoteListeningOn":{"port":37801},"osReleaseId":"ubuntu","arch":"x86_64","webUiAccessToken":"","sshAuthSock":"","display":"","tmpDir":"/run/user/41042","platform":"linux","connectionToken":"a11a111a-1a11-11aa-1a1a-11a11aa111aa","pid":83938,"ipcHandlePath":"/var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-2ef0d9ce17bbe074fbc7bf86195d17bc1a93c36e.sock","socksPort":52193,"startupTime":1635555338990}
[19:57:04.938] Running server is stale. Ignoring
[19:57:04.939] Using commit id "34eeebdd66033633ac22efd5fab928ae37916168" and quality "insider" for server
[19:57:04.942] Install and start server if needed
[19:57:04.954] PATH: /Users/thechargedneutron/.rbenv/shims:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/thechargedneutron/Library/Android/sdk/emulator:/Users/thechargedneutron/Library/Android/sdk/tools:/Users/thechargedneutron/Library/Android/sdk/tools/bin:/Users/thechargedneutron/Library/Android/sdk/platform-tools
[19:57:04.954] Checking ssh with "ssh -V"
[19:57:04.963] > OpenSSH_8.1p1, LibreSSL 2.7.3

[19:57:04.964] askpass server listening on /var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-243fffb2716a3c88297ba544bff67b3ee7beb0c2.sock
[19:57:04.964] Spawning local server with {"serverId":1,"ipcHandlePath":"/var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-0dabf9d3e7ca1a73ca37809a7142fdfd3c525eb0.sock","sshCommand":"ssh","sshArgs":["-v","-T","-D","52243","-o","ConnectTimeout=15","`hostname`"],"dataFilePath":"/Users/thechargedneutron/Library/Application Support/Code - Insiders/User/globalStorage/ms-vscode-remote.remote-ssh/vscode-ssh-host-76b40f39-34eeebdd66033633ac22efd5fab928ae37916168-0.65.8/data.json"}
[19:57:04.964] Local server env: {"DISPLAY":"1","ELECTRON_RUN_AS_NODE":"1","SSH_ASKPASS":"/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/local-server/askpass.sh","VSCODE_SSH_ASKPASS_NODE":"/Applications/Visual Studio Code - Insiders.app/Contents/Frameworks/Code - Insiders Helper (Renderer).app/Contents/MacOS/Code - Insiders Helper (Renderer)","VSCODE_SSH_ASKPASS_MAIN":"/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/askpass-main.js","VSCODE_SSH_ASKPASS_HANDLE":"/var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-243fffb2716a3c88297ba544bff67b3ee7beb0c2.sock"}
[19:57:04.967] Spawned 83978
[19:57:05.101] > local-server-1> Spawned ssh, pid=83983
[19:57:05.102] stderr> OpenSSH_8.1p1, LibreSSL 2.7.3
[19:57:05.324] stderr> debug1: Server host key: ecdsa-sha2-nistp256 SHA256:n09TbVQEP2lfMtVM1hdZAsZs5lfm4IakAEmuWgAcfOg
[19:57:05.620] Got askpass request: {"request":"Enter passphrase for key '/Users/thechargedneutron/.ssh/id_rsa':"}
[19:57:05.621] Detected passphrase message
[19:57:05.621] Listening for interwindow password on /var/folders/bn/wchw2lf52zdc_y9p6bm7217m0000gn/T/vscode-ssh-askpass-edfb48cf23ea357f834d86795ca6b6fb713bbc2c.sock
[19:57:05.621] Writing password prompt to globalState
[19:57:10.883] Passphrase dialog canceled
[19:57:10.883] Interactor signaled cancel
[19:57:10.884] Cleaning up other-window auth server
[19:57:10.903] stderr> `username@hostname`: Permission denied (publickey).
[19:57:10.904] > local-server-1> ssh child died, shutting down
[19:57:10.906] Local server exit: 0
[19:57:10.906] Received install output: local-server-1> Spawned ssh, pid=83983
OpenSSH_8.1p1, LibreSSL 2.7.3
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:n09TbVQEP2lfMtVM1hdZAsZs5lfm4IakAEmuWgAcfOg
`username@hostname`: Permission denied (publickey).
local-server-1> ssh child died, shutting down

[19:57:10.908] Resolver error: Error: Permission denied (publickey).
	at Function.Create (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:397041)
	at /Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:395235
	at Object.t.handleInstallOutput (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:395620)
	at Object.e [as tryInstallWithLocalServer] (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:435292)
	at processTicksAndRejections (internal/process/task_queues.js:93:5)
	at async /Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:452665
	at async Object.t.withShowDetailsEvent (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:456460)
	at async /Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:433816
	at async C (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:430392)
	at async Object.t.resolveWithLocalServer (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:433431)
	at async Object.t.resolve (/Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:454065)
	at async /Users/thechargedneutron/.vscode-insiders/extensions/ms-vscode-remote.remote-ssh-0.65.8/out/extension.js:1:526942
[19:57:10.910] ------

@roblourens
Copy link
Member

Well, the socket might close for some reason in the time that the laptop was asleep. Seems that sometimes ssh in the terminal will survive a brief sleep, sometimes not, same for vscode. Also, we implement our own timeout where once the remote is not reachable for some amount of time, we assume it's disconnected and try to reconnect. So I have to just call this "by design". @alexdima I wonder whether it could be improved for scenarios like this, if we can check the previous socket again after waking from sleep.

It's much more smooth if you set up SSH with your passphrase in the ssh agent so you don't need to type anything to connect

@alexdima
Copy link
Member

alexdima commented Nov 2, 2021

@roblourens The core will invoke the resolver again. You can definitely return the same port again from the resolver if the previous SSH tunnel is still healthy. For example, once it started the server, Remote WSL always returns the same port from the resolver.

@thechargedneutron
Copy link
Author

Honestly, I think there should be at least this level of robustness to keep the session alive for cases when the laptop has just gone to sleep. If I set my screen sleep time to 10 minutes the ssh remains active for this long. But if I set my screen sleep to 1 minute the ssh disconnects as soon as the screen goes off even for a few seconds. I appreciate the your efforts in discussing this issue. Thanks!

@roblourens
Copy link
Member

We have the ability to test the connection again, but wouldn't it make more sense for the core to do that?

@alexdima
Copy link
Member

alexdima commented Nov 3, 2021

We have the ability to test the connection again, but wouldn't it make more sense for the core to do that?

Doing that in all cases means that in many cases we would waste around 30s (the timeout we have) trying to connect to a possibly broken resolver port. So we always just invoke the resolver which can trivially return the same port if it so wishes.

e.g. of what you could do in the SSH resolver (if you want to):

class Resolver {
	private lastPort: number = 0;
	private lastPortReusedTime: number = 0;

	public resolve(ctx) {
		const shouldReuseLastPort = (
			this.lastPort
			&& (Date.now() - this.lastPortReusedTime > 5 * 60 * 1000) // the port was not reused within the past 5 minutes
		);
		if (shouldReuseLastPort /* && tunnel process still alive */) {
			// try to reuse the same tunnel, maybe it still works fine
			this.lastPortReusedTime = Date.now();
			return this.lastPort;
		}
		this.lastPortReusedTime = 0;
		// ...
		// this.lastPort = ...
		return this.lastPort;
	}
}

The core calling the resolver gives the most freedom. Resolvers that know that they are really disconnected can recreate the connection, while resolvers which don't know can try to be lucky and return first the old port and only on a subsequent resolve call create a new tunnel.

What do you think?

@roblourens
Copy link
Member

If we reuse the port, and it turns out to be stuck, will vscode spend 30s trying to connect to it? Because that will happen sometimes. The resolver can also ping the server through the port with a shorter timeout.

@alexdima
Copy link
Member

alexdima commented Nov 3, 2021

If we reuse the port, and it turns out to be stuck, will vscode spend 30s trying to connect to it? Because that will happen sometimes. The resolver can also ping the server through the port with a shorter timeout.

👍 Yes, that is exactly the point I was trying to make. When the core tries to reconnect to a host:port provided by the resolver, it sets a 30s timeout. If the timeout fires without the connection succeeding, it goes again to the resolver to resolve again and so on...

@roblourens roblourens self-assigned this Nov 3, 2021
@roblourens roblourens added feature-request Request for new features or functionality and removed info-needed Issue requires more information from poster labels Nov 3, 2021
@roblourens roblourens added this to the Backlog milestone Nov 3, 2021
@roblourens roblourens reopened this Nov 3, 2021
@roblourens
Copy link
Member

@tanhakabir I've been wanting to do an optimization like this. We can ping the /delay-shutdown endpoint on the previous resolved port to check whether it's still responding, similar to what the install script does. If it responds, we can reuse the port. But if the resolver is invoked again within a short amount of time we would assume it's not healthy and open a new connection.

@tanhakabir tanhakabir modified the milestones: Backlog, On Deck, November 2021 Nov 12, 2021
@jamestut
Copy link

jamestut commented Nov 14, 2021

OK so I have the same symptoms, but a different configuration. In a nutshell, my configuration is like:

SSH on mac host to 127.0.0.1 -> Unix socket -> VSOCK -> Linux' guest SSH.

  • VSCode Version: 1.61
  • Local OS Version: macOS 11.6.1
  • Remote OS Version: Fedora 35

As the connection is performed over VSOCK, it is guaranteed to be stable (no disconnects). I have tested this using both SSH, netcat, and socat on terminal, then put my MacBook to sleep, and all of them are stills connected upon waking up the MacBook. However, VSCode immediately ask for reloading the window.

One thing to note is that the time on the VM momentarily drifts backwards for about 5 seconds before it syncs back to the host time. Also for what its worth, this is a custom hypervisor based on macOS Hypervisor.framework.

@jamestut
Copy link

jamestut commented Jan 3, 2023

Hi all, the problem on macOS is that Chrome is the one to blame: Chrome will disconnect active WebSocket connection, even the one made locally to 127.0.0.1 (e.g. the extension's internal proxy), when it detects that the mac is put to sleep. We can simulate trigger the system sleep right away on macOS by using the pmset sleepnow command (this will also instantly trigger Chrome to drop its connection). Any TCP connections from other apps including SSH to localhost should continue to work and not dropped, but that's not the case with Chrome.

Also FWIW, Safari does not do this.

Not sure about Windows as I don't have a Windows machine for the time being, but I think on Windows it is not related to the Chrome's problem I mentioned AFAIK.

@jamestut
Copy link

Hi @kieferrm @eleanorjboyd @roblourens, apologise beforehand for tagging. However, considering that this is Google Chrome's issue of cutting off WebSocket connection when the laptop is put to standby, I wonder what are your approach in tackling this problem?

Are you going to patch the underlying Electron code on the upcoming VSCode to fix this behaviour, or are you going to make this extension be able to persist states between reconnection?

@roblourens
Copy link
Member

I wouldn't mess with that, we just need to make sure that we are reconnecting cleanly when waking up again. We haven't had time to look into this though. Thanks for the tip about Chrome though!

@movy
Copy link

movy commented Feb 20, 2023

Not exactly related to disconnect-on-sleep issue, but just an UX improvement idea: currently VSCode allows at most 8 reconnect attempts before it gives up and requires window reload, is it possible to set max to 20 attempts for example? Disconnections can be hard to avoid for people on the move, and when it comes to reconnection I observe vastly different behaviour between VPS providers, some of them allow faster reconnection while with others it takes forever, especially when VPN is involved. I guess it all boils down to their routers config and is beyond our control, but to make VSCode more resilient, probably trying reconnecting for a bit longer is sufficient in such scenarios.

image

@eleanorjboyd
Copy link
Member

eleanorjboyd commented Feb 21, 2023

Hi @movy, happy to talk more in depth on this if you create another issue but if you set the max reconnection attempts to null it actually tries many more times than 8 (somewhere in the hundreds). We should update that setting to reflect that but as a fix for now to get more attempts just set that value to null- issue to track that is here: #7552. Thanks

@shrijayan
Copy link

OK so I have the same symptoms, but a different configuration. In a nutshell, my configuration is like:

SSH on mac host to 127.0.0.1 -> Unix socket -> VSOCK -> Linux' guest SSH.

  • VSCode Version: 1.61
  • Local OS Version: macOS 11.6.1
  • Remote OS Version: Fedora 35

As the connection is performed over VSOCK, it is guaranteed to be stable (no disconnects). I have tested this using both SSH, netcat, and socat on terminal, then put my MacBook to sleep, and all of them are stills connected upon waking up the MacBook. However, VSCode immediately ask for reloading the window.

One thing to note is that the time on the VM momentarily drifts backwards for about 5 seconds before it syncs back to the host time. Also for what its worth, this is a custom hypervisor based on macOS Hypervisor.framework.

How to do this any step-by-step guide. I am trying to connect SSH server using VS Code but when my laptop goes to sleep the process that runs stops and the server disconnects again I need to start from first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature-request Request for new features or functionality ssh Issue in vscode-remote SSH
Projects
None yet
Development

No branches or pull requests