Using non root user in controller

[Himavanth]

Using a different user than root would be helpful in mitigating security risks that could arise because of root privileges.
Have done some basic testing.
Would like some feedback.

Related issue and scope

My changes affect the following components

• API
• [ x] Controller
• Message Bus (e.g., Kafka)
• Loadbalancer
• Invoker
• Intrinsic actions (e.g., sequences, conductors)
• Data stores (e.g., CouchDB)
• Tests
• Deployment
• CLI
• General tooling
• Documentation

Types of changes

• Bug fix (generally a non-breaking change which closes an issue).
• Enhancement or new feature (adds new functionality).
• [x ] Breaking change (a bug fix or enhancement which changes existing behavior).

Checklist:

• [ x] I signed an Apache CLA.
• I reviewed the style guides and followed the recommendations (Travis CI will check :).
• I added tests to cover my changes.
• My changes require further changes to the documentation.
• I updated the documentation where necessary.

[markusthoemmes]

Which security risks are you talking about specifically? Note that we setup docker to use usernamespaces. The root user in the container does not equal to a root user on the system itself.

[Himavanth]

Got it. I will check the namespace config if that serves my purpose (say prevent socket access and thereby prevent it from being able to start other docker containers).
Meanwhile i see this best practice from docker about using a non-root user. https://docs.docker.com/develop/develop-images/dockerfile_best-practices/#user
Do you think it makes sense here?

[mcdan]

I think doing this will break DockerClientWithFileAccess because it relies on the user who is running the invoker's ability to read the log files written by the docker daemon, which is of course running as root.

[fmaschler]

I checked the namespaces but on the host the process still runs as root:

$ docker exec -i -t controller0 /bin/bash
bash-4.3# ps                                                                                                                                                                                 
PID   USER     TIME   COMMAND
    1 root       5:32 /usr/lib/jvm/java-8-oracle/bin/java -Djava.security.egd=file:/dev/./urandom -Xmx2g -XX:+
bash-4.3# exit
$ ps -aux | grep java.security
root     21745  9.2  6.7 5736432 545532 ?      Ssl  13:46   5:54 /usr/lib/jvm/java-8-oracle/bin/java -Djava.security.egd=file:/dev/./urandom -Xmx2g -XX:+...

When I run the changed image it already fails at deployment while copying jmxremote to /root/

@mcdan Do you mean the container logs? Cause they are owned by the non-root user. If the invoker reads the daemon log at /var/log/upstart/docker.log it feels like a security issue anyway.

[mcdan]

@fmaschler okay, sounds good.

[fmaschler]

Is there any update on this?

[sechunOH]

@fmaschler
This may not right but I think user namespace remapping is applied to invoker machines.
So you can see argument 'userns=host' in invoker ansible deployment script.
But Runtime containers may be spawned by invoker are using usernamespace remap, so root privileged processes in runtime containers are not really root privileged on the invoker host.

[fmaschler]

I'm not an expert in namespaces but I see what you mean. Though those container processes run as root on the host. Even if they may not have the same privilege inside the container this should be changed if there is no reason about it.

[sechunOH]

@fmaschler
not processes as root on the host, but as root in containers.
It is for preventing privilege-escalation attacks from within a container.

https://docs.docker.com/engine/security/userns-remap/

[fmaschler]

This is good when you talk about INSIDE containers but this is about the OUTSIDE: The containers process on the host runs as root.

The best way to prevent privilege-escalation attacks from within a container is to configure your container’s applications to run as unprivileged users. For containers whose processes must run as the root user within the container, you can re-map this user to a less-privileged user on the Docker host.

The question is if controller, kafka, invoker and the runtime containers all need to run as root?

[rabbah]

@Himavanth @fmaschler using this discussion, we will open an issue to document security-related best practices for an openhisk deployment. In this case, creating namespace so that root on the host is not root inside the container.

See #3746.

[fmaschler]

Great @rabbah! Hopefully this will speed up the discussion and solve the problem in #3603.

[Himavanth]

@rabbah @fmaschler My 2 cents.. I have found that both the approaches have their limitations.
Using a non-root user within the container works for controller and action containers but does not work for invoker since invoker needs privileged access to create action containers.

A User namespace has its own limitations documented here. https://docs.docker.com/engine/security/userns-remap/#user-namespace-known-limitations
One of the limitations listed is pid=host which is the default ansible config in OW Invoker.

The best practice as per docker seems to " configure your container’s applications to run as unprivileged users"

[codecov-io]

Codecov Report

Merging #3579 into master will decrease coverage by 4.92%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #3579      +/-   ##
==========================================
- Coverage   86.05%   81.12%   -4.93%     
==========================================
  Files         147      147              
  Lines        7162     7162              
  Branches      457      457              
==========================================
- Hits         6163     5810     -353     
- Misses        999     1352     +353

Impacted Files	Coverage Δ
...core/database/cosmosdb/RxObservableImplicits.scala	0% <0%> (-100%)	⬇️
...core/database/cosmosdb/CosmosDBArtifactStore.scala	0% <0%> (-95.52%)	⬇️
...sk/core/database/cosmosdb/CosmosDBViewMapper.scala	0% <0%> (-92.6%)	⬇️
...whisk/core/database/cosmosdb/CosmosDBSupport.scala	0% <0%> (-83.34%)	⬇️
...abase/cosmosdb/CosmosDBArtifactStoreProvider.scala	0% <0%> (-58.83%)	⬇️
...la/whisk/core/database/cosmosdb/CosmosDBUtil.scala	92% <0%> (-4%)	⬇️
...rc/main/scala/whisk/common/ForcibleSemaphore.scala	88.46% <0%> (-3.85%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 16c2c53...e093881. Read the comment docs.

[chetanmeh]

Can this be more restrictive? Which part requires the permissions to be open up in conntroller

[Himavanth]

The jmx files are moved to this dir at init stage and then their permissions are changed to 600 by copyJMXFiles.sh. This is the only permission combination that i found working. There are no other files under root folder. If required we could create a different folder just to store these files.

[chetanmeh]

May be change its owner to NOT_ROOT_USER or make it 640 i.e. avoid execute bit

[chetanmeh]

From discussion I understand that proper setting up of namespace would avoid change user for process. However as @Himavanth mentioned that its a good practice to follow per docker docs

If a service can run without privileges, use USER to change to a non-root user

Unless there is any cons/downside in doing this change we should make this change. @markusthoemmes @rabbah @ddragosd Thoughts?

[chetanmeh]

Ping!

[markusthoemmes]

LGTM, @chetanmeh if you think this doesn't break anything, feel free to merge.

[rabbah]

@chetanmeh OK with me.

[Himavanth]

There is a PG fail that needs to be debugged here. Following is the stacktrace. Thx @vvraskin for this.
Error: Password file not found: /root/jmxremote.password
sun.management.AgentConfigurationError
at sun.management.jmxremote.ConnectorBootstrap.checkPasswordFile(ConnectorBootstrap.java:563)
at sun.management.jmxremote.ConnectorBootstrap.startRemoteConnectorServer(ConnectorBootstrap.java:426)
at sun.management.Agent.startAgent(Agent.java:262)
at sun.management.Agent.startAgent(Agent.java:452)
mv: can't stat '/root/jmxremote.access': Permission denied
mv: can't stat '/root/jmxremote.password': Permission denied
chmod: /root/jmxremote.*: Permission denied

[Himavanth]

Tracking this at #4012

[vvraskin]

LGTM, thanks for addressing the comments as per our slack discussion!