Upgrading from 2023.10.7 to 2024.2.3 with argocd, failed with redis template #271

thinkhead · 2024-06-15T01:02:46Z

Describe the bug
Upgrading from 2023.10.7 to 2024.2.3 with argocd, failed with redis template.

Relevant info
Kube version: v1.26.13+rke2r1
ArgoCD: v2.10.12+cb6f5ac
Authentik Helm Chart Version: 2024.2.3
Deployment: [helm]

Logs
Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = helm template . --name-template authentik-rke-dev --namespace authentik-rke-dev --kube-version 1.26 --values /tmp/23a262ae-25f2-47e6-92dc-b9f146fb464e --include-crds failed exit status 1: Error: YAML parse error on authentik/charts/redis/templates/master/application.yaml: error converting YAML to JSON: yaml: line 40: mapping values are not allowed in this context Use --debug flag to render out invalid YAML

To Reproduce
Upgrading from 2023.10.7 with this argocd application:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
  name: authentik
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: tools
  destination:
    namespace: authentik-rke-dev
    name: rke-dev
  source:
    repoURL: 'https://charts.goauthentik.io'
    targetRevision: 2023.10.7
    chart: authentik
    helm:
      values: |
        redis:
          enabled: true
        replicas: 3
        server:
          replicas: 3
        ingress:
          enabled: true
          annotations:
            kubernetes.io/ingress.class: nginx
          hosts:
            - host: xxxx
              paths:
                - path: "/"
                  pathType: Prefix
          tls:
            - secretName: xxxxx-tls
              hosts:
                - xxxxx
        image:
          pullSecrets:
            - name: 'image-pull-secret'
        worker:
          replicas: 3
        geoip:
          enabled: true
          accountId: "xxxxx"
          licenseKey: "xxxx"
        authentik:
          secret_key: "xxxx"
          error_reporting:
            enabled: false
          postgresql:
            password: "xxxxx"
        prometheus:
          rules:
            create: true
          serviceMonitor:
            create: true
        postgresql:
          enabled: true
          postgresqlPassword: "xxxxxx"
  syncPolicy:
    automated: 
      prune: true 
      selfHeal: true 
      allowEmpty: false 
    syncOptions: 
    - CreateNamespace=true
    retry:
      limit: 0

To 2024.2.3

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  annotations:
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
  name: authentik
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io
spec:
  project: tools
  destination:
    namespace: authentik-rke-dev
    name: rke-dev
  source:
    repoURL: 'https://charts.goauthentik.io'
    targetRevision: 2024.2.3
    chart: authentik
    helm:
      values: |
        redis:
          enabled: true
        server:
          serviceMonitor:
            enabled: true
          replicas: 3
          ingress:
            enabled: true
            annotations:
              kubernetes.io/ingress.class: nginx
            hosts:
              - xxxxxx
            paths:
              - /
            pathType: Prefix
            tls:
              - secretName: xxxxx-tls
                hosts:
                  - xxxxx
        global:
          imagePullSecrets:
            - name: 'image-pull-secret'
          revisionHistoryLimit: 3
        worker:
          replicas: 3
        geoip:
          enabled: true
          accountId: "****"
          licenseKey: "***"
        authentik:
          secret_key: "********"
          postgresql:
            password: "********"
        prometheus:
          rules:
            enabled: true
        postgresql:
          enabled: true
          auth:
            password: "**********"
          primary:
            persistence:
              enabled: true
              storageClass: longhorn
              accessModes:
                - ReadWriteOnce
  syncPolicy:
    automated: 
      prune: true 
      selfHeal: true 
      allowEmpty: false 
    syncOptions: 
    - CreateNamespace=true
    retry:
      limit: 0

This gave me the following error in argocd and prevent further upgrade:

Failed to load target state: failed to generate manifest for source 1 of 1: rpc error: code = Unknown desc = `helm template . --name-template authentik-rke-dev --namespace authentik-rke-dev --kube-version 1.26 --values /tmp/23a262ae-25f2-47e6-92dc-b9f146fb464e <api versions removed> --include-crds` failed exit status 1: Error: YAML parse error on authentik/charts/redis/templates/master/application.yaml: error converting YAML to JSON: yaml: line 40: mapping values are not allowed in this context Use --debug flag to render out invalid YAML

It's seem to pushing this template, but i didn't find any useful information

< apiVersion: apps/v1
< kind: StatefulSet
< metadata:
<   annotations:
<     kubectl.kubernetes.io/last-applied-configuration: |
<       {"apiVersion":"apps/v1","kind":"StatefulSet","metadata":{"annotations":{},"labels":{"app.kubernetes.io/component":"master","app.kubernetes.io/instance":"authentik-rke-dev","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"redis","helm.sh/chart":"redis-15.7.6"},"name":"authentik-rke-dev-redis-master","namespace":"authentik-rke-dev"},"spec":{"replicas":1,"selector":{"matchLabels":{"app.kubernetes.io/component":"master","app.kubernetes.io/instance":"authentik-rke-dev","app.kubernetes.io/name":"redis"}},"serviceName":"authentik-rke-dev-redis-headless","template":{"metadata":{"annotations":{"checksum/configmap":"e3d798c2426b7e8af3b7ff62bc75c42fa2b2ce0b9697f80b0541425cf93515d2","checksum/health":"d1c98f37a2bd9bdeca53a6d909e0a29fb5fd21aea4f49db97fafcfdfce7260c4","checksum/scripts":"1fabf9e118ae712e8080d52a3043b52b069a64171519025774fff78f0bfeda30","checksum/secret":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"},"labels":{"app.kubernetes.io/component":"master","app.kubernetes.io/instance":"authentik-rke-dev","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"redis","helm.sh/chart":"redis-15.7.6"}},"spec":{"affinity":{"nodeAffinity":null,"podAffinity":null,"podAntiAffinity":{"preferredDuringSchedulingIgnoredDuringExecution":[{"podAffinityTerm":{"labelSelector":{"matchLabels":{"app.kubernetes.io/component":"master","app.kubernetes.io/instance":"authentik-rke-dev","app.kubernetes.io/name":"redis"}},"namespaces":["authentik-rke-dev"],"topologyKey":"kubernetes.io/hostname"},"weight":1}]}},"containers":[{"args":["-c","/opt/bitnami/scripts/start-scripts/start-master.sh"],"command":["/bin/bash"],"env":[{"name":"BITNAMI_DEBUG","value":"false"},{"name":"REDIS_REPLICATION_MODE","value":"master"},{"name":"ALLOW_EMPTY_PASSWORD","value":"yes"},{"name":"REDIS_TLS_ENABLED","value":"no"},{"name":"REDIS_PORT","value":"6379"}],"image":"docker.io/bitnami/redis:6.2.10-debian-11-r13","imagePullPolicy":"IfNotPresent","livenessProbe":{"exec":{"command":["sh","-c","/health/ping_liveness_local.sh 5"]},"failureThreshold":5,"initialDelaySeconds":20,"periodSeconds":5,"successThreshold":1,"timeoutSeconds":6},"name":"redis","ports":[{"containerPort":6379,"name":"redis"}],"readinessProbe":{"exec":{"command":["sh","-c","/health/ping_readiness_local.sh 1"]},"failureThreshold":5,"initialDelaySeconds":20,"periodSeconds":5,"successThreshold":1,"timeoutSeconds":2},"resources":{"limits":{},"requests":{}},"securityContext":{"runAsUser":1001},"volumeMounts":[{"mountPath":"/opt/bitnami/scripts/start-scripts","name":"start-scripts"},{"mountPath":"/health","name":"health"},{"mountPath":"/data","name":"redis-data","subPath":null},{"mountPath":"/opt/bitnami/redis/mounted-etc","name":"config"},{"mountPath":"/opt/bitnami/redis/etc/","name":"redis-tmp-conf"},{"mountPath":"/tmp","name":"tmp"}]}],"securityContext":{"fsGroup":1001},"serviceAccountName":"authentik-rke-dev-redis","terminationGracePeriodSeconds":30,"volumes":[{"configMap":{"defaultMode":493,"name":"authentik-rke-dev-redis-scripts"},"name":"start-scripts"},{"configMap":{"defaultMode":493,"name":"authentik-rke-dev-redis-health"},"name":"health"},{"configMap":{"name":"authentik-rke-dev-redis-configuration"},"name":"config"},{"emptyDir":{},"name":"redis-tmp-conf"},{"emptyDir":{},"name":"tmp"}]}},"updateStrategy":{"rollingUpdate":{},"type":"RollingUpdate"},"volumeClaimTemplates":[{"metadata":{"labels":{"app.kubernetes.io/component":"master","app.kubernetes.io/instance":"authentik-rke-dev","app.kubernetes.io/name":"redis"},"name":"redis-data"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"8Gi"}}}}]}}
<   generation: 3
<   labels:
<     app.kubernetes.io/component: master
<     app.kubernetes.io/instance: authentik-rke-dev
<     app.kubernetes.io/managed-by: Helm
<     app.kubernetes.io/name: redis
<     helm.sh/chart: redis-15.7.6
<   managedFields:
<   - apiVersion: apps/v1
<     fieldsType: FieldsV1
<     fieldsV1:
<       f:metadata:
<         f:annotations:
<           .: {}
<           f:kubectl.kubernetes.io/last-applied-configuration: {}
<         f:labels:
<           .: {}
<           f:app.kubernetes.io/component: {}
<           f:app.kubernetes.io/instance: {}
<           f:app.kubernetes.io/managed-by: {}
<           f:app.kubernetes.io/name: {}
<           f:helm.sh/chart: {}
<       f:spec:
<         f:podManagementPolicy: {}
<         f:revisionHistoryLimit: {}
<         f:selector: {}
<         f:serviceName: {}
<         f:template:
<           f:metadata:
<             f:annotations:
<               .: {}
<               f:checksum/configmap: {}
<               f:checksum/health: {}
<               f:checksum/scripts: {}
<               f:checksum/secret: {}
<             f:labels:
<               .: {}
<               f:app.kubernetes.io/component: {}
<               f:app.kubernetes.io/instance: {}
<               f:app.kubernetes.io/managed-by: {}
<               f:app.kubernetes.io/name: {}
<               f:helm.sh/chart: {}
<           f:spec:
<             f:affinity:
<               .: {}
<               f:podAntiAffinity:
<                 .: {}
<                 f:preferredDuringSchedulingIgnoredDuringExecution: {}
<             f:containers:
<               k:{"name":"redis"}:
<                 .: {}
<                 f:args: {}
<                 f:command: {}
<                 f:env:
<                   .: {}
<                   k:{"name":"ALLOW_EMPTY_PASSWORD"}:
<                     .: {}
<                     f:name: {}
<                     f:value: {}
<                   k:{"name":"BITNAMI_DEBUG"}:
<                     .: {}
<                     f:name: {}
<                     f:value: {}
<                   k:{"name":"REDIS_PORT"}:
<                     .: {}
<                     f:name: {}
<                     f:value: {}
<                   k:{"name":"REDIS_REPLICATION_MODE"}:
<                     .: {}
<                     f:name: {}
<                     f:value: {}
<                   k:{"name":"REDIS_TLS_ENABLED"}:
<                     .: {}
<                     f:name: {}
<                     f:value: {}
<                 f:image: {}
<                 f:imagePullPolicy: {}
<                 f:livenessProbe:
<                   .: {}
<                   f:exec:
<                     .: {}
<                     f:command: {}
<                   f:failureThreshold: {}
<                   f:initialDelaySeconds: {}
<                   f:periodSeconds: {}
<                   f:successThreshold: {}
<                   f:timeoutSeconds: {}
<                 f:name: {}
<                 f:ports:
<                   .: {}
<                   k:{"containerPort":6379,"protocol":"TCP"}:
<                     .: {}
<                     f:containerPort: {}
<                     f:name: {}
<                     f:protocol: {}
<                 f:readinessProbe:
<                   .: {}
<                   f:exec:
<                     .: {}
<                     f:command: {}
<                   f:failureThreshold: {}
<                   f:initialDelaySeconds: {}
<                   f:periodSeconds: {}
<                   f:successThreshold: {}
<                   f:timeoutSeconds: {}
<                 f:resources: {}
<                 f:securityContext:
<                   .: {}
<                   f:runAsUser: {}
<                 f:terminationMessagePath: {}
<                 f:terminationMessagePolicy: {}
<                 f:volumeMounts:
<                   .: {}
<                   k:{"mountPath":"/data"}:
<                     .: {}
<                     f:mountPath: {}
<                     f:name: {}
<                   k:{"mountPath":"/health"}:
<                     .: {}
<                     f:mountPath: {}
<                     f:name: {}
<                   k:{"mountPath":"/opt/bitnami/redis/etc/"}:
<                     .: {}
<                     f:mountPath: {}
<                     f:name: {}
<                   k:{"mountPath":"/opt/bitnami/redis/mounted-etc"}:
<                     .: {}
<                     f:mountPath: {}
<                     f:name: {}
<                   k:{"mountPath":"/opt/bitnami/scripts/start-scripts"}:
<                     .: {}
<                     f:mountPath: {}
<                     f:name: {}
<                   k:{"mountPath":"/tmp"}:
<                     .: {}
<                     f:mountPath: {}
<                     f:name: {}
<             f:dnsPolicy: {}
<             f:restartPolicy: {}
<             f:schedulerName: {}
<             f:securityContext:
<               .: {}
<               f:fsGroup: {}
<             f:serviceAccount: {}
<             f:serviceAccountName: {}
<             f:terminationGracePeriodSeconds: {}
<             f:volumes:
<               .: {}
<               k:{"name":"config"}:
<                 .: {}
<                 f:configMap:
<                   .: {}
<                   f:defaultMode: {}
<                   f:name: {}
<                 f:name: {}
<               k:{"name":"health"}:
<                 .: {}
<                 f:configMap:
<                   .: {}
<                   f:defaultMode: {}
<                   f:name: {}
<                 f:name: {}
<               k:{"name":"redis-tmp-conf"}:
<                 .: {}
<                 f:emptyDir: {}
<                 f:name: {}
<               k:{"name":"start-scripts"}:
<                 .: {}
<                 f:configMap:
<                   .: {}
<                   f:defaultMode: {}
<                   f:name: {}
<                 f:name: {}
<               k:{"name":"tmp"}:
<                 .: {}
<                 f:emptyDir: {}
<                 f:name: {}
<         f:updateStrategy:
<           f:rollingUpdate:
<             .: {}
<             f:partition: {}
<           f:type: {}
<         f:volumeClaimTemplates: {}
<     manager: argocd-controller
<     operation: Update
<     time: "2024-06-14T19:25:28Z"
<   - apiVersion: apps/v1
<     fieldsType: FieldsV1
<     fieldsV1:
<       f:status:
<         f:availableReplicas: {}
<         f:collisionCount: {}
<         f:currentReplicas: {}
<         f:currentRevision: {}
<         f:observedGeneration: {}
<         f:readyReplicas: {}
<         f:replicas: {}
<         f:updateRevision: {}
<         f:updatedReplicas: {}
<     manager: kube-controller-manager
<     operation: Update
<     subresource: status
<     time: "2024-06-14T21:02:20Z"
<   name: authentik-rke-dev-redis-master
<   namespace: authentik-rke-dev
<   resourceVersion: "378141239"
<   uid: 0d784fc1-b9f8-4dcb-a0f7-66cd4ea1051f
< spec:
<   podManagementPolicy: OrderedReady
<   replicas: 1
<   revisionHistoryLimit: 10
<   selector:
<     matchLabels:
<       app.kubernetes.io/component: master
<       app.kubernetes.io/instance: authentik-rke-dev
<       app.kubernetes.io/name: redis
<   serviceName: authentik-rke-dev-redis-headless
<   template:
<     metadata:
<       annotations:
<         checksum/configmap: e3d798c2426b7e8af3b7ff62bc75c42fa2b2ce0b9697f80b0541425cf93515d2
<         checksum/health: d1c98f37a2bd9bdeca53a6d909e0a29fb5fd21aea4f49db97fafcfdfce7260c4
<         checksum/scripts: 1fabf9e118ae712e8080d52a3043b52b069a64171519025774fff78f0bfeda30
<         checksum/secret: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
<       creationTimestamp: null
<       labels:
<         app.kubernetes.io/component: master
<         app.kubernetes.io/instance: authentik-rke-dev
<         app.kubernetes.io/managed-by: Helm
<         app.kubernetes.io/name: redis
<         helm.sh/chart: redis-15.7.6
<     spec:
<       affinity:
<         podAntiAffinity:
<           preferredDuringSchedulingIgnoredDuringExecution:
<           - podAffinityTerm:
<               labelSelector:
<                 matchLabels:
<                   app.kubernetes.io/component: master
<                   app.kubernetes.io/instance: authentik-rke-dev
<                   app.kubernetes.io/name: redis
<               namespaces:
<               - authentik-rke-dev
<               topologyKey: kubernetes.io/hostname
<             weight: 1
<       containers:
<       - args:
<         - -c
<         - /opt/bitnami/scripts/start-scripts/start-master.sh
<         command:
<         - /bin/bash
<         env:
<         - name: BITNAMI_DEBUG
<           value: "false"
<         - name: REDIS_REPLICATION_MODE
<           value: master
<         - name: ALLOW_EMPTY_PASSWORD
<           value: "yes"
<         - name: REDIS_TLS_ENABLED
<           value: "no"
<         - name: REDIS_PORT
<           value: "6379"
<         image: docker.io/bitnami/redis:6.2.10-debian-11-r13
<         imagePullPolicy: IfNotPresent
<         livenessProbe:
<           exec:
<             command:
<             - sh
<             - -c
<             - /health/ping_liveness_local.sh 5
<           failureThreshold: 5
<           initialDelaySeconds: 20
<           periodSeconds: 5
<           successThreshold: 1
<           timeoutSeconds: 6
<         name: redis
<         ports:
<         - containerPort: 6379
<           name: redis
<           protocol: TCP
<         readinessProbe:
<           exec:
<             command:
<             - sh
<             - -c
<             - /health/ping_readiness_local.sh 1
<           failureThreshold: 5
<           initialDelaySeconds: 20
<           periodSeconds: 5
<           successThreshold: 1
<           timeoutSeconds: 2
<         resources: {}
<         securityContext:
<           runAsUser: 1001
<         terminationMessagePath: /dev/termination-log
<         terminationMessagePolicy: File
<         volumeMounts:
<         - mountPath: /opt/bitnami/scripts/start-scripts
<           name: start-scripts
<         - mountPath: /health
<           name: health
<         - mountPath: /data
<           name: redis-data
<         - mountPath: /opt/bitnami/redis/mounted-etc
<           name: config
<         - mountPath: /opt/bitnami/redis/etc/
<           name: redis-tmp-conf
<         - mountPath: /tmp
<           name: tmp
<       dnsPolicy: ClusterFirst
<       restartPolicy: Always
<       schedulerName: default-scheduler
<       securityContext:
<         fsGroup: 1001
<       serviceAccount: authentik-rke-dev-redis
<       serviceAccountName: authentik-rke-dev-redis
<       terminationGracePeriodSeconds: 30
<       volumes:
<       - configMap:
<           defaultMode: 493
<           name: authentik-rke-dev-redis-scripts
<         name: start-scripts
<       - configMap:
<           defaultMode: 493
<           name: authentik-rke-dev-redis-health
<         name: health
<       - configMap:
<           defaultMode: 420
<           name: authentik-rke-dev-redis-configuration
<         name: config
<       - emptyDir: {}
<         name: redis-tmp-conf
<       - emptyDir: {}
<         name: tmp
<   updateStrategy:
<     rollingUpdate:
<       partition: 0
<     type: RollingUpdate
<   volumeClaimTemplates:
<   - apiVersion: v1
<     kind: PersistentVolumeClaim
<     metadata:
<       creationTimestamp: null
<       labels:
<         app.kubernetes.io/component: master
<         app.kubernetes.io/instance: authentik-rke-dev
<         app.kubernetes.io/name: redis
<       name: redis-data
<     spec:
<       accessModes:
<       - ReadWriteOnce
<       resources:
<         requests:
<           storage: 8Gi
<       volumeMode: Filesystem
<     status:
<       phase: Pending
< status:
<   availableReplicas: 1
<   collisionCount: 0
<   currentReplicas: 1
<   currentRevision: authentik-rke-dev-redis-master-856b54c949
<   observedGeneration: 3
<   readyReplicas: 1
<   replicas: 1
<   updateRevision: authentik-rke-dev-redis-master-856b54c949
<   updatedReplicas: 1

Removing redis unblock the upgrade, but the server is looking for redis in loop and failed to start

{"event": "Redis Connection failed, retrying... (Error -3 connecting to authentik-rke-dev-redis-master:6379. Temporary failure in name resolution.)", "level": "info", "logger": "authentik.lib.config", "timestamp": 1718336817.1424649, "redis_url": "redis:https://:@authentik-rke-dev-redis-master:6379/0"}
{"event": "Redis Connection failed, retrying... (Error -3 connecting to authentik-rke-dev-redis-master:6379. Temporary failure in name resolution.)", "level": "info", "logger": "authentik.lib.config", "timestamp": 1718336818.1951334, "redis_url": "redis:https://:@authentik-rke-dev-redis-master:6379/0"}

The text was updated successfully, but these errors were encountered:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrading from 2023.10.7 to 2024.2.3 with argocd, failed with redis template #271

Upgrading from 2023.10.7 to 2024.2.3 with argocd, failed with redis template #271

thinkhead commented Jun 15, 2024

Upgrading from 2023.10.7 to 2024.2.3 with argocd, failed with redis template #271

Upgrading from 2023.10.7 to 2024.2.3 with argocd, failed with redis template #271

Comments

thinkhead commented Jun 15, 2024