readinessProbe not failing if command is not installed

[AlbertoSoutullo]

What happened?

I have the following readinessProbe in my .yaml file:

readinessProbe:
  exec:
    command:
      - /bin/sh
      - -c
      - >
        curl -s http:https://127.0.0.1:8008/metrics |
        awk '!/^#/ && /^libp2p_gossipsub_healthy_peers_topics /{
          print "Found gossipsub:", $0;
          if ($2 == 1.0) {
            exit 0;  # success, healthy state
          } else {
            exit 1;  # failure, unhealthy state
          }
        }'
  successThreshold: 5
  initialDelaySeconds: 5
  periodSeconds: 1
  failureThreshold: 2
  timeoutSeconds: 5

The pod only has one container, and this container doesn't have curl installed. Still, the readinessProbe is making the pod as Ready.

I then manually entered in the container, and installed curl by hand. 1 second after that, the POD was marked as not ready.

If I don't manually install curl, and I run the same command, I got an error, as curl is not installed.

What did you expect to happen?

The POD should not be marked as ready.

How can we reproduce it (as minimally and precisely as possible)?

I edited my yaml to be reproducible and reach to the same issue.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nodes
  namespace: zerotesting
spec:
  replicas: 1
  podManagementPolicy: "Parallel"
  selector:
    matchLabels:
      app: zerotenkay
  template:
    metadata:
      labels:
        app: zerotenkay
    spec:
      containers:
        - name: test
          image: alpine:3.18
          imagePullPolicy: IfNotPresent
          readinessProbe:
            exec:
              command:
                - /bin/sh
                - -c
                - >
                  curl -s http:https://127.0.0.1:8008/metrics |
                  awk 'asd'
            successThreshold: 5
            initialDelaySeconds: 5
            periodSeconds: 1
            failureThreshold: 2
            timeoutSeconds: 5
          resources:
            requests:
              memory: "64Mi"
              cpu: "150m"
            limits:
              memory: "600Mi"
              cpu: "400m"
          command:
            - sh
            - -c
            - sleep 99999

If I run this, I still can see that the pod is marked as Ready, even curl is not installed.

Anything else we need to know?

No response

Kubernetes version

$ kubectl version
 v1.28.9+k3s1

Cloud provider

Own deployment

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
# paste output here

# On Windows:
C:\> wmic os get Caption, Version, BuildNumber, OSArchitecture
# paste output here

Install tools

Container runtime (CRI) and version (if applicable)

Related plugins (CNI, CSI, ...) and versions (if applicable)

[k8s-ci-robot]

There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[mauri870]

I think that is expected. You are piping the output to awk and it returns status code 0 regardless of the failure in the previous command, readinessProbe uses the status == 0 to consider the command a success.

If you want it to fail regardless of the pipe operation you can update your command with set -e; set -o pipefail

[T-Lakshmi]

/kind support

[AlbertoSoutullo]

I think that is expected. You are piping the output to awk and it returns status code 0 regardless of the failure in the previous command, readinessProbe uses the status == 0 to consider the command a success.

If you want it to fail regardless of the pipe operation you can update your command with set -e; set -o pipefail

Indeed. I was expecting that the status return wouldn't be zero, actually. Closing the issue, it was a misunderstanding from my side.