Hanging at TASK [k3s_agent : Manage k3s service]

[lukababu]

Not able to install the playbook

Expected Behavior

Installation to be successful

Current Behavior

It should not hang at:

TASK [k3s_agent : Manage k3s service]

and instead, it should tell me what the issue is at the very least.

The log output where it hangs give the following output:

TASK [k3s_agent : Manage k3s service] ************************************************************************************************************
task path: /Users/luke/Documents/GitHub/k3s-ansible/roles/k3s_agent/tasks/main.yml:31
<10.1.1.84> ESTABLISH SSH CONNECTION FOR USER: serveradmin
<10.1.1.84> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/ff535aba42"' 10.1.1.84 '/bin/sh -c '"'"'echo ~serveradmin && sleep 0'"'"''
<10.1.1.85> ESTABLISH SSH CONNECTION FOR USER: serveradmin
<10.1.1.85> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/760d370fbc"' 10.1.1.85 '/bin/sh -c '"'"'echo ~serveradmin && sleep 0'"'"''
<10.1.1.85> (0, b'/home/serveradmin\n', b"OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.85 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/luke/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/luke/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69157\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<10.1.1.84> (0, b'/home/serveradmin\n', b"OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.84 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/luke/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/luke/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69156\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<10.1.1.85> ESTABLISH SSH CONNECTION FOR USER: serveradmin
<10.1.1.84> ESTABLISH SSH CONNECTION FOR USER: serveradmin
<10.1.1.85> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/760d370fbc"' 10.1.1.85 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/serveradmin/.ansible/tmp `"&& mkdir "` echo /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116 `" && echo ansible-tmp-1709506509.344173-69233-54000031077116="` echo /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116 `" ) && sleep 0'"'"''
<10.1.1.84> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/ff535aba42"' 10.1.1.84 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/serveradmin/.ansible/tmp `"&& mkdir "` echo /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441 `" && echo ansible-tmp-1709506509.3442528-69232-149471937072441="` echo /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441 `" ) && sleep 0'"'"''
<10.1.1.84> (0, b'ansible-tmp-1709506509.3442528-69232-149471937072441=/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441\n', b"OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.84 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/luke/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/luke/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69156\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<10.1.1.85> (0, b'ansible-tmp-1709506509.344173-69233-54000031077116=/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116\n', b"OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.85 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/luke/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/luke/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69157\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
Using module file /opt/homebrew/Cellar/ansible/9.3.0/libexec/lib/python3.12/site-packages/ansible/modules/systemd.py
Using module file /opt/homebrew/Cellar/ansible/9.3.0/libexec/lib/python3.12/site-packages/ansible/modules/systemd.py
<10.1.1.85> PUT /Users/luke/.ansible/tmp/ansible-local-67430nlu55g4i/tmpby5gstl2 TO /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/AnsiballZ_systemd.py
<10.1.1.84> PUT /Users/luke/.ansible/tmp/ansible-local-67430nlu55g4i/tmpnvoqdrtv TO /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/AnsiballZ_systemd.py
<10.1.1.85> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/760d370fbc"' '[10.1.1.85]'
<10.1.1.84> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/ff535aba42"' '[10.1.1.84]'
<10.1.1.85> (0, b'sftp> put /Users/luke/.ansible/tmp/ansible-local-67430nlu55g4i/tmpby5gstl2 /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/AnsiballZ_systemd.py\n', b'OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.85 is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/Users/luke/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/Users/luke/.ssh/known_hosts2\'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69157\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 2\r\ndebug2: Server supports extension "[email protected]" revision 2\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug3: Sent message [email protected] I:1\r\ndebug3: Received limits reply T:201 I:1\r\ndebug1: Using server download size 261120\r\ndebug1: Using server upload size 261120\r\ndebug1: Server handle limit 1019; using 64\r\ndebug2: Sending SSH2_FXP_REALPATH "."\r\ndebug3: Sent message fd 3 T:16 I:2\r\ndebug3: SSH2_FXP_REALPATH . -> /home/serveradmin\r\ndebug3: Looking up /Users/luke/.ansible/tmp/ansible-local-67430nlu55g4i/tmpby5gstl2\r\ndebug2: Sending SSH2_FXP_STAT "/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/AnsiballZ_systemd.py"\r\ndebug3: Sent message fd 3 T:17 I:3\r\ndebug1: stat remote: No such file or directory\r\ndebug2: do_upload: upload local "/Users/luke/.ansible/tmp/ansible-local-67430nlu55g4i/tmpby5gstl2" to remote "/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/AnsiballZ_systemd.py"\r\ndebug2: Sending SSH2_FXP_OPEN "/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/AnsiballZ_systemd.py"\r\ndebug3: Sent dest message SSH2_FXP_OPEN I:4 P:/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/AnsiballZ_systemd.py M:0x001a\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:0 S:261120\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 261120 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:261120 S:44322\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 44322 bytes at 261120\r\ndebug3: Sent message SSH2_FXP_CLOSE I:5\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.1.1.84> (0, b'sftp> put /Users/luke/.ansible/tmp/ansible-local-67430nlu55g4i/tmpnvoqdrtv /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/AnsiballZ_systemd.py\n', b'OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.84 is address\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts\' -> \'/Users/luke/.ssh/known_hosts\'\r\ndebug3: expanded UserKnownHostsFile \'~/.ssh/known_hosts2\' -> \'/Users/luke/.ssh/known_hosts2\'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69156\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug2: Remote version: 3\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 2\r\ndebug2: Server supports extension "[email protected]" revision 2\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug2: Server supports extension "[email protected]" revision 1\r\ndebug3: Sent message [email protected] I:1\r\ndebug3: Received limits reply T:201 I:1\r\ndebug1: Using server download size 261120\r\ndebug1: Using server upload size 261120\r\ndebug1: Server handle limit 1019; using 64\r\ndebug2: Sending SSH2_FXP_REALPATH "."\r\ndebug3: Sent message fd 3 T:16 I:2\r\ndebug3: SSH2_FXP_REALPATH . -> /home/serveradmin\r\ndebug3: Looking up /Users/luke/.ansible/tmp/ansible-local-67430nlu55g4i/tmpnvoqdrtv\r\ndebug2: Sending SSH2_FXP_STAT "/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/AnsiballZ_systemd.py"\r\ndebug3: Sent message fd 3 T:17 I:3\r\ndebug1: stat remote: No such file or directory\r\ndebug2: do_upload: upload local "/Users/luke/.ansible/tmp/ansible-local-67430nlu55g4i/tmpnvoqdrtv" to remote "/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/AnsiballZ_systemd.py"\r\ndebug2: Sending SSH2_FXP_OPEN "/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/AnsiballZ_systemd.py"\r\ndebug3: Sent dest message SSH2_FXP_OPEN I:4 P:/home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/AnsiballZ_systemd.py M:0x001a\r\ndebug3: Sent message SSH2_FXP_WRITE I:6 O:0 S:261120\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 6 261120 bytes at 0\r\ndebug3: Sent message SSH2_FXP_WRITE I:7 O:261120 S:44324\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: In write loop, ack for 7 44324 bytes at 261120\r\ndebug3: Sent message SSH2_FXP_CLOSE I:5\r\ndebug3: SSH2_FXP_STATUS 0\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n')
<10.1.1.85> ESTABLISH SSH CONNECTION FOR USER: serveradmin
<10.1.1.84> ESTABLISH SSH CONNECTION FOR USER: serveradmin
<10.1.1.85> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/760d370fbc"' 10.1.1.85 '/bin/sh -c '"'"'chmod u+x /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/ /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/AnsiballZ_systemd.py && sleep 0'"'"''
<10.1.1.84> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/ff535aba42"' 10.1.1.84 '/bin/sh -c '"'"'chmod u+x /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/ /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/AnsiballZ_systemd.py && sleep 0'"'"''
<10.1.1.85> (0, b'', b"OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.85 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/luke/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/luke/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69157\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<10.1.1.84> (0, b'', b"OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.84 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/luke/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/luke/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 69156\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
<10.1.1.85> ESTABLISH SSH CONNECTION FOR USER: serveradmin
<10.1.1.84> ESTABLISH SSH CONNECTION FOR USER: serveradmin
<10.1.1.85> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/760d370fbc"' -tt 10.1.1.85 '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=ppmcclyynuohvwxrqqpjgmftzmwpgbiw] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-ppmcclyynuohvwxrqqpjgmftzmwpgbiw ; /usr/bin/python3 /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.344173-69233-54000031077116/AnsiballZ_systemd.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
<10.1.1.84> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/ff535aba42"' -tt 10.1.1.84 '/bin/sh -c '"'"'sudo -H -S -p "[sudo via ansible, key=qmvwstjyfnuuzuoyfsrjirtzqzyrefhe] password:" -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-qmvwstjyfnuuzuoyfsrjirtzqzyrefhe ; /usr/bin/python3 /home/serveradmin/.ansible/tmp/ansible-tmp-1709506509.3442528-69232-149471937072441/AnsiballZ_systemd.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
Escalation succeeded

Steps to Reproduce

1. Run ansible-playbook ./site.yml -i ./inventory/my-cluster/hosts.ini -bK
2. Hangs at the TASK [k3s_agent : Manage k3s service]
3. Run ansible-playbook ./reset.yml -i ./inventory/my-cluster/hosts.ini
4. Run ansible-playbook ./reboot.yml -i ./inventory/my-cluster/hosts.ini
5. Re-run ansible-playbook ./site.yml -i ./inventory/my-cluster/hosts.ini -bK, same error occurs

Context (variables)

Running playbook from:

Operating system: Ventura 13.5.1 (22G90)
Hardware: Macbook Pro M2 Max

Deploying in:

Operating system: Proxmox cloud-based Ubuntu VM clones 22.04 amd64 (Jammy)
Hardware: 4 Core, 4 GB RAM, 64 GB NVMe Drive.

Variables Used

all.yml

---
k3s_version: v1.29.0+k3s1
# this is the user that has ssh access to these machines
ansible_user: serveradmin
systemd_dir: /etc/systemd/system

# Set your timezone
system_timezone: "Asia/Tbilisi"

# interface which will be used for flannel
flannel_iface: "eth0"

# uncomment calico_iface to use tigera operator/calico cni instead of flannel https://docs.tigera.io/calico/latest/about
# calico_iface: "eth0"
calico_ebpf: false           # use eBPF dataplane instead of iptables
calico_tag: "v3.27.0"        # calico version tag

# uncomment cilium_iface to use cilium cni instead of flannel or calico
# ensure v4.19.57, v5.1.16, v5.2.0 or more recent kernel
# cilium_iface: "eth0"
cilium_mode: "native"        # native when nodes on same subnet or using bgp, else set routed
cilium_tag: "v1.14.6"        # cilium version tag
cilium_hubble: true          # enable hubble observability relay and ui

# if using calico or cilium, you may specify the cluster pod cidr pool
cluster_cidr: "10.52.0.0/16"

# enable cilium bgp control plane for lb services and pod cidrs. disables metallb.
cilium_bgp: false

# bgp parameters for cilium cni. only active when cilium_iface is defined and cilium_bgp is true.
cilium_bgp_my_asn: "64513"
cilium_bgp_peer_asn: "64512"
cilium_bgp_peer_address: "192.168.30.1"
cilium_bgp_lb_cidr: "192.168.31.0/24"   # cidr for cilium loadbalancer ipam

# apiserver_endpoint is virtual ip-address which will be configured on each master
apiserver_endpoint: "10.1.1.222"

# k3s_token is required  masters can talk together securely
# this token should be alpha numeric only
k3s_token: "32_CHAR_ALPHA_NUMERIC_STRING"

# The IP on which the node is reachable in the cluster.
# Here, a sensible default is provided, you can still override
# it for each of your hosts, though.
k3s_node_ip: "{{ ansible_facts[(cilium_iface | default(calico_iface | default(flannel_iface)))]['ipv4']['address'] }}"

# Disable the taint manually by setting: k3s_master_taint = false
k3s_master_taint: "{{ true if groups['node'] | default([]) | length >= 1 else false }}"

# these arguments are recommended for servers as well as agents:
extra_args: >-
  {{ '--flannel-iface=' + flannel_iface if calico_iface is not defined and cilium_iface is not defined else '' }}
  --node-ip={{ k3s_node_ip }}

# change these to your liking, the only required are: --disable servicelb, --tls-san {{ apiserver_endpoint }}
# the contents of the if block is also required if using calico or cilium
extra_server_args: >-
  {{ extra_args }}
  {{ '--node-taint node-role.kubernetes.io/master=true:NoSchedule' if k3s_master_taint else '' }}
  {% if calico_iface is defined or cilium_iface is defined %}
  --flannel-backend=none
  --disable-network-policy
  --cluster-cidr={{ cluster_cidr | default('10.52.0.0/16') }}
  {% endif %}
  --tls-san {{ apiserver_endpoint }}
  --disable servicelb
  --disable traefik

extra_agent_args: >-
  {{ extra_args }}

# image tag for kube-vip
kube_vip_tag_version: "v0.6.4"

# tag for kube-vip-cloud-provider manifest
# kube_vip_cloud_provider_tag_version: "main"

# kube-vip ip range for load balancer
# (uncomment to use kube-vip for services instead of MetalLB)
# kube_vip_lb_ip_range: "192.168.30.80-192.168.30.90"

# metallb type frr or native
metal_lb_type: "native"

# metallb mode layer2 or bgp
metal_lb_mode: "layer2"

# bgp options
# metal_lb_bgp_my_asn: "64513"
# metal_lb_bgp_peer_asn: "64512"
# metal_lb_bgp_peer_address: "192.168.30.1"

# image tag for metal lb
metal_lb_speaker_tag_version: "v0.13.12"
metal_lb_controller_tag_version: "v0.13.12"

# metallb ip range for load balancer
metal_lb_ip_range: "10.1.1.70-10.1.1.80"

# Only enable if your nodes are proxmox LXC nodes, make sure to configure your proxmox nodes
# in your hosts.ini file.
# Please read https://gist.github.com/triangletodd/02f595cd4c0dc9aac5f7763ca2264185 before using this.
# Most notably, your containers must be privileged, and must not have nesting set to true.
# Please note this script disables most of the security of lxc containers, with the trade off being that lxc
# containers are significantly more resource efficient compared to full VMs.
# Mixing and matching VMs and lxc containers is not supported, ymmv if you want to do this.
# I would only really recommend using this if you have particularly low powered proxmox nodes where the overhead of
# VMs would use a significant portion of your available resources.
proxmox_lxc_configure: false
# the user that you would use to ssh into the host, for example if you run ssh some-user@my-proxmox-host,
# set this value to some-user
proxmox_lxc_ssh_user: root
# the unique proxmox ids for all of the containers in the cluster, both worker and master nodes
proxmox_lxc_ct_ids:
  - 181
  - 182
  - 183
  - 184
  - 185

# Only enable this if you have set up your own container registry to act as a mirror / pull-through cache
# (harbor / nexus / docker's official registry / etc).
# Can be beneficial for larger dev/test environments (for example if you're getting rate limited by docker hub),
# or air-gapped environments where your nodes don't have internet access after the initial setup
# (which is still needed for downloading the k3s binary and such).
# k3s's documentation about private registries here: https://docs.k3s.io/installation/private-registry
custom_registries: false
# The registries can be authenticated or anonymous, depending on your registry server configuration.
# If they allow anonymous access, simply remove the following bit from custom_registries_yaml
#   configs:
#     "registry.domain.com":
#       auth:
#         username: yourusername
#         password: yourpassword
# The following is an example that pulls all images used in this playbook through your private registries.
# It also allows you to pull your own images from your private registry, without having to use imagePullSecrets
# in your deployments.
# If all you need is your own images and you don't care about caching the docker/quay/ghcr.io images,
# you can just remove those from the mirrors: section.
custom_registries_yaml: |
  mirrors:
    docker.io:
      endpoint:
        - "https://registry.domain.com/v2/dockerhub"
    quay.io:
      endpoint:
        - "https://registry.domain.com/v2/quayio"
    ghcr.io:
      endpoint:
        - "https://registry.domain.com/v2/ghcrio"
    registry.domain.com:
      endpoint:
        - "https://registry.domain.com"

  configs:
    "registry.domain.com":
      auth:
        username: yourusername
        password: yourpassword

# Only enable and configure these if you access the internet through a proxy
# proxy_env:
#   HTTP_PROXY: "http:https://proxy.domain.local:3128"
#   HTTPS_PROXY: "http:https://proxy.domain.local:3128"
#   NO_PROXY: "*.domain.local,127.0.0.0/8,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16"

Possible Solution

• I've checked the General Troubleshooting Guide
• I've looked into and analyzed the logs
• I utilized ChatGPT
• I searched online for similar issues
• My system time on my Mac is different from the time zone I am deploying it to
• Delete nodes and do a reinstall, re-clone branch and setup fresh

[lukababu]

I was able to get 4/5 nodes working after "Delete nodes and do a reinstall, re-clone branch and setup fresh". A possible culprit might have been resizing the partition on the nodes, or installing net-tools package.

Checking the log output the following lines appear right after:

2024-03-03 17:01:06,965 p=79858 u=luke n=ansible | Escalation succeeded
2024-03-03 17:01:06,965 p=79858 u=luke n=ansible | Escalation succeeded
2024-03-03 17:02:49,568 p=79858 u=luke n=ansible | <10.1.1.84> (0, b'\r\n{"name": "k3s-node", "changed": true, "status": {"Type": "notify", "Restart": "always", "NotifyAccess": "main", "RestartUSec": "5s", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "1min 30s", "TimeoutAbortUSec": "1min 30s", "TimeoutStartFailureMode": "terminate", "TimeoutStopFailureMode": "terminate", "RuntimeMaxUSec": "infinity", "WatchdogUSec": "infinity", "WatchdogTimestamp": "n/a", "WatchdogTimestampMonotonic": "0", "RootDirectoryStartOnly": "no", "RemainAfterExit": "no", "GuessMainPID": "yes", "MainPID": "0", "ControlPID": "0", "FileDescriptorStoreMax": "0", "NFileDescriptorStore": "0", "StatusErrno": "0", "Result": "success", "ReloadResult": "success", "CleanResult": "success", "UID": "[not set]", "GID": "[not set]", "NRestarts": "0", "OOMPolicy": "continue", "ExecMainStartTimestamp": "n/a", "ExecMainStartTimestampMonotonic": "0", "ExecMainExitTimestamp": "n/a", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainCode": "0", "ExecMainStatus": "0", "ExecStartPre": "{ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; ignore_errors=yes ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPreEx": "{ path=/sbin/modprobe ; argv[]=/sbin/modprobe overlay ; flags=ignore-failure ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/local/bin/k3s ; argv[]=/usr/local/bin/k3s agent --server https://10.1.1.222:6443 --token K10c713ed47480e52638ab99b6acd3f4d1f22f3ed66f65355ed09cc75deb347d5ee::server:unoEjdr9aXGRhWVczytGUKjyEF3tFDYz --flannel-iface=eth0 --node-ip=10.1.1.84 ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartEx": "{ path=/usr/local/bin/k3s ; argv[]=/usr/local/bin/k3s agent --server https://10.1.1.222:6443 --token K10c713ed47480e52638ab99b6acd3f4d1f22f3ed66f65355ed09cc75deb347d5ee::server:unoEjdr9aXGRhWVczytGUKjyEF3tFDYz --flannel-iface=eth0 --node-ip=10.1.1.84 ; flags= ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "Slice": "system.slice", "MemoryCurrent": "[not set]", "MemoryAvailable": "infinity", "CPUUsageNSec": "[not set]", "TasksCurrent": "[not set]", "IPIngressBytes": "[no data]", "IPIngressPackets": "[no data]", "IPEgressBytes": "[no data]", "IPEgressPackets": "[no data]", "IOReadBytes": "18446744073709551615", "IOReadOperations": "18446744073709551615", "IOWriteBytes": "18446744073709551615", "IOWriteOperations": "18446744073709551615", "Delegate": "yes", "DelegateControllers": "cpu cpuacct cpuset io blkio memory devices pids bpf-firewall bpf-devices bpf-foreign bpf-socket-bind", "CPUAccounting": "yes", "CPUWeight": "[not set]", "StartupCPUWeight": "[not set]", "CPUShares": "[not set]", "StartupCPUShares": "[not set]", "CPUQuotaPerSecUSec": "infinity", "CPUQuotaPeriodUSec": "infinity", "IOAccounting": "no", "IOWeight": "[not set]", "StartupIOWeight": "[not set]", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "StartupBlockIOWeight": "[not set]", "MemoryAccounting": "yes", "DefaultMemoryLow": "0", "DefaultMemoryMin": "0", "MemoryMin": "0", "MemoryLow": "0", "MemoryHigh": "infinity", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MemoryLimit": "infinity", "DevicePolicy": "auto", "TasksAccounting": "yes", "TasksMax": "infinity", "IPAccounting": "no", "ManagedOOMSwap": "auto", "ManagedOOMMemoryPressure": "auto", "ManagedOOMMemoryPressureLimit": "0", "ManagedOOMPreference": "none", "UMask": "0022", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitNOFILE": "1048576", "LimitNOFILESoft": "1048576", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitSIGPENDING": "15347", "LimitSIGPENDINGSoft": "15347", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "OOMScoreAdjust": "0", "CoredumpFilter": "0x33", "Nice": "0", "IOSchedulingClass": "2", "IOSchedulingPriority": "4", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUAffinityFromNUMA": "no", "NUMAPolicy": "n/a", "TimerSlackNSec": "50000", "CPUSchedulingResetOnFork": "no", "NonBlocking": "no", "StandardInput": "null", "StandardOutput": "journal", "StandardError": "inherit", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "SyslogPriority": "30", "SyslogLevelPrefix": "yes", "SyslogLevel": "6", "SyslogFacility": "3", "LogLevelMax": "-1", "LogRateLimitIntervalUSec": "0", "LogRateLimitBurst": "0", "SecureBits": "0", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend cap_audit_read cap_perfmon cap_bpf cap_checkpoint_restore", "DynamicUser": "no", "RemoveIPC": "no", "PrivateTmp": "no", "PrivateDevices": "no", "ProtectClock": "no", "ProtectKernelTunables": "no", "ProtectKernelModules": "no", "ProtectKernelLogs": "no", "ProtectControlGroups": "no", "PrivateNetwork": "no", "PrivateUsers": "no", "PrivateMounts": "no", "PrivateIPC": "no", "ProtectHome": "no", "ProtectSystem": "no", "SameProcessGroup": "no", "UtmpMode": "init", "IgnoreSIGPIPE": "yes", "NoNewPrivileges": "no", "SystemCallErrorNumber": "2147483646", "LockPersonality": "no", "RuntimeDirectoryPreserve": "no", "RuntimeDirectoryMode": "0755", "StateDirectoryMode": "0755", "CacheDirectoryMode": "0755", "LogsDirectoryMode": "0755", "ConfigurationDirectoryMode": "0755", "TimeoutCleanUSec": "infinity", "MemoryDenyWriteExecute": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "RestrictNamespaces": "no", "MountAPIVFS": "no", "KeyringMode": "private", "ProtectProc": "default", "ProcSubset": "all", "ProtectHostname": "no", "KillMode": "process", "KillSignal": "15", "RestartKillSignal": "15", "FinalKillSignal": "9", "SendSIGKILL": "yes", "SendSIGHUP": "no", "WatchdogSignal": "6", "Id": "k3s-node.service", "Names": "k3s-node.service", "Requires": "sysinit.target system.slice", "Conflicts": "shutdown.target", "Before": "shutdown.target", "After": "network-online.target basic.target sysinit.target systemd-journald.socket system.slice", "Documentation": "https://k3s.io", "Description": "Lightweight Kubernetes", "LoadState": "loaded", "ActiveState": "inactive", "FreezerState": "running", "SubState": "dead", "FragmentPath": "/etc/systemd/system/k3s-node.service", "UnitFileState": "disabled", "UnitFilePreset": "enabled", "StateChangeTimestamp": "n/a", "StateChangeTimestampMonotonic": "0", "InactiveExitTimestamp": "n/a", "InactiveExitTimestampMonotonic": "0", "ActiveEnterTimestamp": "n/a", "ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestamp": "n/a", "ActiveExitTimestampMonotonic": "0", "InactiveEnterTimestamp": "n/a", "InactiveEnterTimestampMonotonic": "0", "CanStart": "yes", "CanStop": "yes", "CanReload": "no", "CanIsolate": "no", "CanFreeze": "yes", "StopWhenUnneeded": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "AllowIsolate": "no", "DefaultDependencies": "yes", "OnSuccessJobMode": "fail", "OnFailureJobMode": "replace", "IgnoreOnIsolate": "no", "NeedDaemonReload": "no", "JobTimeoutUSec": "infinity", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "ConditionResult": "no", "AssertResult": "no", "ConditionTimestamp": "n/a", "ConditionTimestampMonotonic": "0", "AssertTimestamp": "n/a", "AssertTimestampMonotonic": "0", "Transient": "no", "Perpetual": "no", "StartLimitIntervalUSec": "10s", "StartLimitBurst": "5", "StartLimitAction": "none", "FailureAction": "none", "SuccessAction": "none", "CollectMode": "inactive"}, "enabled": true, "state": "started", "invocation": {"module_args": {"name": "k3s-node", "daemon_reload": true, "state": "restarted", "enabled": true, "daemon_reexec": false, "scope": "system", "no_block": false, "force": null, "masked": null}}}\r\n', b"OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.84 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/luke/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/luke/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81724\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\nShared connection to 10.1.1.84 closed.\r\n")
2024-03-03 17:02:49,572 p=79858 u=luke n=ansible | <10.1.1.84> ESTABLISH SSH CONNECTION FOR USER: serveradmin
2024-03-03 17:02:49,574 p=79858 u=luke n=ansible | <10.1.1.84> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="serveradmin"' -o ConnectTimeout=10 -o 'ControlPath="/Users/luke/.ansible/cp/ff535aba42"' 10.1.1.84 '/bin/sh -c '"'"'rm -f -r /home/serveradmin/.ansible/tmp/ansible-tmp-1709510466.4454558-81800-273274775614732/ > /dev/null 2>&1 && sleep 0'"'"''
2024-03-03 17:02:49,667 p=79858 u=luke n=ansible | <10.1.1.84> (0, b'', b"OpenSSH_9.0p1, LibreSSL 3.3.6\r\ndebug1: Reading configuration data /Users/luke/.ssh/config\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 21: include /etc/ssh/ssh_config.d/* matched no files\r\ndebug1: /etc/ssh/ssh_config line 54: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.1.1.84 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/Users/luke/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/Users/luke/.ssh/known_hosts2'\r\ndebug1: Authenticator provider $SSH_SK_PROVIDER did not resolve; disabling\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 81724\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")
2024-03-03 17:02:49,681 p=79858 u=luke n=ansible | changed: [10.1.1.84] => {
    "changed": true,

Let me know if I should attach the full log, in case that is helpful!