stanley_rsa is owned by root:root instead of stanley user in k8s

[cmmdrdata]

in the st2client pod sudo su - stanley then try to login somewhere using stanley_rsa key. It won't work unless you are root or use sudo.

whoami
stanley
stanley@stackstorm-st2client-b96dd9f76-t9kjp:~$ ls -l ~stanley/.ssh/
total 0
lrwxrwxrwx 1 root root 18 Sep  9 20:35 stanley_rsa -> ..data/stanley_rsa
stanley@stackstorm-st2client-b96dd9f76-t9kjp:~$ ssh undercloud.admin -i ~stanley/.ssh/stanley_rsa
The authenticity of host 'undercloud.admin' (10.75.163.57)' can't be established.
ECDSA key fingerprint is SHA256:Nr3mtGvtxhNNMeCzpy642VkASYji/u0xESuTTVKe4.
Are you sure you want to continue connecting (yes/no)? yes
Failed to add the host to the list of known hosts (/home/stanley/.ssh/known_hosts).
        ******** WARNING: UNAUTHORIZED PERSONS, DO NOT PROCEED ********
This system is intended to be used solely by authorized users in the course of
legitimate corporate business.  Users are monitored to the extent necessary to
properly administer the system, to identify the unauthorized users or users
operating beyone their proper authroity, and to investigate improper access or
use.  By accessing the system, you are consenting to this monitoring.
Additionally, users accessing this system agree that they understand and will
comply with all Verizon Information  Security and Privacy policies, including
policy statements, instructions, standards and guidelines.
        ******** WARNING: UNAUTHORIZED PERSONS, DO NOT PROCEED ********

Load key "/home/stanley/.ssh/stanley_rsa": Permission denied
[email protected]'s password:

[arm4b]

Thanks for the report.
This looks like a bug indeed.

If you're motivated to submit a fix, there are 2 places in code to update:
https://github.com/StackStorm/stackstorm-ha/blob/93fe06e3c84cf9bd97a711246ad1b8e5d5624503/templates/deployments.yaml#L1014-L1021
and
https://github.com/StackStorm/stackstorm-ha/blob/93fe06e3c84cf9bd97a711246ad1b8e5d5624503/templates/deployments.yaml#L1291-L1298

[cmmdrdata]

Oh cool I'd love to. I'm not sure what the exact change is though . Is there an owner: attribute that I can add stanley to or is there some code that also has to read that attribute from the deployment.yaml template file ?

[arm4b]

@cmmdrdata There is no owner attr for K8s volumes, but changing mode to allow reads from others, eg 0444 permission would be sufficient.

[cmmdrdata]

got it, some ssh servers won't let you login though if your key is too open for reading right ?

[arm4b]

That's a good point, I forgot that.

Another thing I just searched for is trying to rely on securityContext.fsGroup to utilize stanley as a group owner. Not sure if that'll work, needs verifying.

[arm4b]

https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#set-the-security-context-for-a-pod

[arm4b]

BTW, after looking at the working stackstorm-ha cluster, st2actionrunner as used by StackStorm has no problems in picking up stanley SSH key as the service is running under root and has different SSH client primitives under the hood.

As a workaround, using ssh undercloud.admin -i /home/stanley/.ssh/stanley_rsa as root should work for you in a meantime.
The right way ideal solution: we indeed should make sure stanley group can read the SSH key.

[cmmdrdata]

security context changed the group ownership on the volume but nt the file :( inside st2actionrunner pod.

ls -l /home/stanley/.ssh/
total 0
lrwxrwxrwx 1 root root 18 Sep 14 16:52 stanley_rsa -> ..data/stanley_rsa

root@stackstorm-st2actionrunner-7888d8ffc7-ml99n:/opt/stackstorm# ls -ld /home/stanley/.ssh/
drwxrwsrwt 3 root stanley 100 Sep 14 16:52 /home/stanley/.ssh/

[arm4b]

Ah, that's really a bummer.

I found this discussion which still seems to be still an issue in K8s: kubernetes/kubernetes#81089

The workaround is pretty hacky: https://stackoverflow.com/questions/49945437/changing-default-file-owner-and-group-owner-of-kubernetes-secrets-files-mounted, - copy a secret to a normal file via intermediate InitContainer, then change its ownership.

[cognifloyd]

I was just looking at adding a postStart hook for a different purpose, but I think it could also be used to chown the ssh key file.

see:
https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/
https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/

It looks like others have done something similar as well:
https://stackoverflow.com/questions/67018408/kubernetes-run-chmod-on-deployment
https://www.jeffgeerling.com/blog/2019/mounting-kubernetes-secret-single-file-inside-pod

[cognifloyd]

#206 fixes this issue. Additional eyes on the fix would be appreciated.