AWS self managed Kubernetes cluster in VPN no internet environment

[panho66]

HI
I have Kubernetes Cluster in AWS but not EKS. EC2 hosts has IAM role could retrieve secret from AWS secret manager without secret/access key. I try IAM roles using annonation and ACCESS/SECRET key in pod environment variable

pods start ok but no secret created.

npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]prestart: [email protected]
npm info lifecycle [email protected]start: [email protected]

[email protected] start /app
./bin/daemon.js
{"level":30,"time":1566036776616,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"loading kube specs","v":1}
{"level":30,"time":1566036776751,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"successfully loaded kube specs","v":1}
{"level":30,"time":1566036776751,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"updating CRD","v":1}
{"level":30,"time":1566036776751,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"Upserting custom resource externalsecrets.kubernetes-client.io","v":1}
{"level":30,"time":1566036776803,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"successfully updated CRD","v":1}
{"level":30,"time":1566036776803,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"starting app","v":1}
Sat, 17 Aug 2019 10:12:56 GMT kubernetes-client deprecated .getStream see https://github.com/godaddy/kubernetes-client/blob/master/merging-with-kubernetes.md at lib/external-secret.js:40:10
{"level":30,"time":1566036776806,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"successfully started app","v":1}
{"level":30,"time":1566036776810,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"spinning up poller {"id":"791fbf92-c0cb-11e9-bd0d-06c6820c1422","namespace":"default","secretDescriptor":{"backendType":"secretManager","data":[{"key":"middleware/rancher/console/blue/dev/admin","name":"password"}],"name":"hello-service"},"ownerReference":{"apiVersion":"kubernetes-client.io/v1","controller":true,"kind":"ExternalSecret","name":"hello-service","uid":"791fbf92-c0cb-11e9-bd0d-06c6820c1422"}}","v":1}
{"level":30,"time":1566036776811,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"starting poller","v":1}
{"level":30,"time":1566036852377,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"stopping and removing poller 791fbf92-c0cb-11e9-bd0d-06c6820c1422","v":1}
{"level":30,"time":1566036852377,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"stopping poller","v":1}
{"level":30,"time":1566036852386,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"spinning up poller {"id":"791fbf92-c0cb-11e9-bd0d-06c6820c1422","namespace":"default","secretDescriptor":{"backendType":"secretManager","data":[{"key":"middleware/rancher/console/blue/dev/admin","name":"password"}],"name":"hello-service"},"ownerReference":{"apiVersion":"kubernetes-client.io/v1","controller":true,"kind":"ExternalSecret","name":"hello-service","uid":"791fbf92-c0cb-11e9-bd0d-06c6820c1422"}}","v":1}
{"level":30,"time":1566036852388,"pid":18,"hostname":"kubernetes-external-secrets-55fc446964-wzvns","msg":"starting poller","v":1}

In EC2 instance, I have to set up proxy for AWS cli to work
export http_proxy=http:https://proxy:8080
export https_proxy=https://proxy:8080
export no_proxy=localhost,127.0.0.1,169.254.169.254

I try to set proxy in the pod environment variable with http_proxy, https_proxy, no_proxy or HTTPS_PROXY, HTTP_PROXY and NO_PROXY

pods did not start correctly

npm info it worked if it ends with ok
npm info using [email protected]
npm info using [email protected]
npm info lifecycle [email protected]prestart: [email protected]
npm info lifecycle [email protected]start: [email protected]

[email protected] start /app
./bin/daemon.js
{"level":30,"time":1566036668226,"pid":17,"hostname":"kubernetes-external-secrets-8f8f4b5b5-w8mlb","msg":"loading kube specs","v":1}
Error: Failed to get /openapi/v2 and /swagger.json: read ECONNRESET
at _getSpec.catch.then.catch.err (/app/node_modules/kubernetes-client/lib/swagger-client.js:58:15)
at process._tickCallback (internal/process/next_tick.js:68:7)
npm info lifecycle [email protected]~start: Failed to exec start script
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: ./bin/daemon.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
npm timing npm Completed in 76105ms
npm ERR! A complete log of this run can be found in:
npm ERR! /home/node/.npm/_logs/2019-08-17T10_12_23_490Z-debug.log

Any idea? Is there a way to enable debug to have more log message to find out why?

[Flydiverny]

Seems like it doesn't manage to reach the kubernetes api to initialize the kubernetes api client.

[ecout]

Your question is posted in a confusing way. It seems contradictory:
1:
"EC2 hosts has IAM role could retrieve secret from AWS secret manager without secret/access key."

2: AWS self managed Kubernetes cluster in VPN no internet environment.

How do you manage to access ANY AWS service from a private Subnet without ANY ROUTES to public? All AWS Services listen on public by default. Try VPC endpoints instead. Somebody else asked the same question a couple of months after you.

[Flydiverny]

Proxy support was added in #601.
Using private clusters seems to work according to #232 and #348