Buggy sycn in ArgoCD

[ivanfoo]

What happened?

We configured the custom ArgocCD Health Checks for Crossplane resources as documented in the official docs here

However, we noticed an odd side effect: Crossplane cannot finish reconciling some resources anymore, it's like ArgoCD was interfering in the middle of the process. It seems to affect only to new created resources, but we are not 100% sure of that.

How can we reproduce it?

For instance, when creating a RDS cluster using a manifest similar to the following:

apiVersion: rds.aws.upbound.io/v1beta1
kind: Cluster
metadata:
  annotations:
    crossplane.io/external-name: an-application-a-database
  name: an-application-a-database
spec:
  deletionPolicy: Orphan
  forProvider:
    applyImmediately: true
    autoGeneratePassword: false
    backupRetentionPeriod: 7
    dbSubnetGroupName: a-subnet-group
    engine: aurora-postgresql
    engineMode: provisioned
    engineVersion: "16.1"
    manageMasterUserPassword: false
    masterPasswordSecretRef:
      key: password
      name: an-application-a-database-rds-master-password
      namespace: a-namespace
    masterUsername: a-master-username
    region: an-aws-region
    serverlessv2ScalingConfiguration:
      - maxCapacity: 4
        minCapacity: 2
    vpcSecurityGroupIds:
      - a-sg-id
  writeConnectionSecretToRef:
    name: an-application-a-database-rds-details
    namespace: a-namespace

You expect Crossplane to generate a secret named an-application-a-database-rds-details with the following keys:

• master_username
• attribute.master.password
• endpoint
• read_endpoint
• port

And the Secret is actually generated. However, the Secret type is Opaque instead of connection.crossplane.io/v1alpha1 , so Crossplane cannot claim ownership of it. That leads the resource to stuck as failed (for ArgoCD and Crossplane too), requiring manual intervention.

We have other resources whose creation process works oddly when the health checks customization is applied on ArgoCD. Everything went back to normal after removing it. We kept the annotation tracking method, though.

Is anybody else suffering a similar problem?

What environment did it happen in?

Crossplane version: 1.16
Providers affected: RDS, SQL, S3 and Gitlab (so far)