Skip to content

Kubewarden certificate rotation issues with ArgoCD

Article Number: 000022158

Environment

Rancher, ArgoCD
Kubewarden-controller v3.1.0

Situation

After upgrading Kubewarden and running it on an RKE2 cluster managed by ArgoCD, the TLS certificate rotates, leading to 'unknown issuer' errors.

At a certain point, ArgoCD detected a change and triggered a controller sync, resulting in the rotation of the secret certificate

{"level":"info","ts":"2025-01-17T08:41:25Z","logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"}

The kubewarden policy-server doesn't always recognize the certificate rotation, causing discrepancies between the certificate presented by the policy server and the one in the secret.

Post "https://policy-server-default.kubewarden.svc:8443/validate/clusterwide-fs-group?timeout=30s": tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "kubewarden-controller-ca")

Cause

The issue is caused by dynamically generating certificates during the Helm chart runtime, which ArgoCD detects as changes, triggering a sync and certificate rotation.After deployment, ArgoCD renders the chart again for comparison, which generates new certificates. The comparison detects differences, making the application out-of-sync right away. If auto-sync with self-heal is enabled, ArgoCD forcefully syncs the chart every 24 hours when the rendered chart cache expires. This behaviour can be simulated at any time by performing a hard refresh of the application from the ArgoCD UI. Since the CA certificate changes, the cert-controller has to rotate the policy server certificate after some time, which eventually causes the issue

Resolution

Modify the ArgoCD Application to include ignoreDifferences to prevent ArgoCD from detecting changes in the dynamically generated certificates.

Add the following configurations to the ArgoCD application, adjusting the namespace as necessary:

  ignoreDifferences:
    - group: ""
      kind: Secret
      name: kubewarden-ca
      namespace: kubewarden-system
      jsonPointers:
        - "/data"
    - group: ""
      kind: Secret
      name: kubewarden-webhook-server-cert
      namespace: kubewarden-system
      jsonPointers:
        - "/data"
    - group: 'admissionregistration.k8s.io'
      kind: 'ValidatingWebhookConfiguration'
      name: 'kubewarden-controller-validating-webhook-configuration'
      jqPathExpressions:
        - '.webhooks[]?.clientConfig.caBundle'
    - group: 'admissionregistration.k8s.io'
      kind: 'MutatingWebhookConfiguration'
      name: 'kubewarden-controller-mutating-webhook-configuration'
      jqPathExpressions:
        - '.webhooks[]?.clientConfig.caBundle'

Restart the policy server after applying the ignoreDifferences configuration to ensure it picks up the new certificate.