OpenSSH sshd disconnects with invalid packet type: 192 on sshpiper >= 1.2.4

[g0dsCookie]

Hi,

I'm facing an issue with sshpiper >= 1.2.4 where sshd receives a packet with type 192 and immediately disconnects afterwards with "connection from 1.2.3.4:11282 closed reason: ssh: disconnect, reason2: Invalid ssh2 packet type: 192".

sshd-backend logs for disconnect:

debug3: receive packet: type 192
debug2: sshpkt_disconnect: sending SSH2_MSG_DISCONNECT: Invalid ssh2 packet type: 192
debug3: send packet: type 1
ssh_dispatch_run_fatal: Connection from user test 10.42.0.59 port 32950: Protocol error

The disconnects happens right after I type something after connecting. Means: Connection works, I get a terminal from the backend but as soon as I type something (press enter, typing a letter) I get disconnected.

SFTP on the other hand works - i can upload/download/delete files, create directories, etc.
Using connection multiplexing with the ControlSocket option and first connecting with SFTP and then multiplexing this connection with SSH I can also use the terminal without disconnects.
So something must be happening when using SSH and trying to send a keystroke for the first time.

With sshpiper 1.2.3 I can't reproduce the error anymore. Digging through the changelog from 1.2.3 to 1.2.4 I found this change in crypto module: tg123/sshpiper.crypto@833695f

Setup

Currently sshpiper is running on a single-node rke2 Kubernetes with this helm chart.
I've installed it with these settings: helm install --create-namespace --namespace sshpiper sshpiper sshpiper/sshpiper --version 0.3.4 --set service.type=LoadBalancer --set service.port=22 --set sshpiper.loglevel=trace --set sshpiper.kubernetes_all_namespaces=true --set rbac.clusterRole=true --set sshpiper.drop_hostkeys_message=true

As backend I'm using a debian:12 base with openssh-9.2p1. I've also tried with lscr.io/linuxserver/openssh-server:latest and got the same disconnects.

I'm not doing anything "fancy" outside of the examples in doc to get the Pipes in k8s working, but I can still post more details about the setup if needed for reproduction.

[tg123]

this is an interesting issue
during handshake, sshpiper replied with ping support to downstream while upstream (sshd) does not support it
opensshd added ping support at version 9.5.

workaround is to upgrade your sshd or use 1.2.3- sshpiper

let me think about how to deal it

[tg123]

btw what is fancy os dist? your ssh client is too new

[g0dsCookie]

btw what is fancy os dist? your ssh client is too new

I'm using Manjaro which already distributes OpenSSH 9.6.

this is an interesting issue during handshake, sshpiper replied with ping support to downstream while upstream (sshd) does not support it opensshd added ping support at version 9.5.

workaround is to upgrade your sshd or use 1.2.3- sshpiper

let me think about how to deal it

If I undestand this correctly the ping@openssh message may either be replied to with the exact copy of the data or may just be ignored.

So one way would be for sshpiper to filter ping@openssh messages if the upstream doesn't support it or reply by itself instead of passing it to upstream.
Though the latter wouldn't reflect the real RTT from backend to upstream which would make the metric kind of irrelevant.

[tg123]

i am not into filtering it as inspecting every single msg contributes to latency

the design of sshpiper is to void touching msg after authentication
however, there was already a filter for msg (drophostkey, off by default)
i am thinking about adding option for ping, but i do not wish to look into every piped msg

[tg123]

chart 0.3.5 published, it can unblock you for now
thanks for reporting this fancy issue