Skip to content

Enforce default HTTPS Rancher certificate with Letsencrypt ECDSA and DNS-01 renewal challenge

Article Number: 000021731

Environment

2.9+

Situation

A self-signed default Rancher certificate is currently used and will be migrated to a stronger Let’s Encrypt ECDSA-386 certificate using the DNS-01 renewal challenge.

Resolution

Before proceeding, make sure you have backups available in case a rollback is needed. Also, ensure you have read and fully understand all the steps before starting.

1 - Modify the URL in the Rancher UI

If the URL changes, modify it under Global settings -> Settings -> server-url.

2 - Configures cert-manager to issue a Let's Encrypt TLS certificate using DNS-01 challenge

If your issuer is Cloudflare, copy this in a file, letsencrypt-cert-manager.yaml for example, and apply it:

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: tls-rancher-ingress
  namespace: cattle-system
spec:
  secretName: tls-rancher-ingress
  commonName: my_rancher.example.com
  dnsNames:
  - my_rancher.example.com
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    email: xxx.yyy@zzz.com
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
    - dns01:
        cloudflare:
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token
---
apiVersion: v1
kind: Secret
data:
  api-token: AAAAAAAAAAAAAAAAAAAAAAAA==
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: cert-manager
type: Opaque

kubectl apply -f letsencrypt-cert-manager.yaml

Note that if you are using other issuers, please check the cert-manager DNS-01 challenge documentation for the correct syntax.

To prevent being locked by Letsencrypt during your tests, use the acme staging environment first.

3 - Reconfigure Rancher

  • Export your previous Rancher helm chart configuration:
helm get values rancher -n cattle-system -o yaml > values.yaml
  • Create a new Helm chart values file with the following settings:
> cat ingress_letsencrypt.yaml
ingress:
  tls:
    source: secret
  extraAnnotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    cert-manager.io/private-key-algorithm: "ECDSA"
    cert-manager.io/private-key-size: "384"
    cert-manager.io/private-key-rotation-policy: "Always"
privateCA: true

Since we want to use ECDSA while the default Rancher Ingress uses RSA, we enforce private key regeneration by setting private-key-rotation-policy to Always. After applying the change and once the new certificate is in place, you can remove the annotation to prevent further private key renewals (See point 8). If you choose to keep RSA as the certificate algorithm, this line is not required.

If you want to stick to your current Rancher version and don't want to update, do not forget to add the --version option followed by your version. Ex: --version=2.10.1

helm upgrade --install rancher rancher-prime/rancher --namespace cattle-system --set hostname=my_rancher.example.com -f values.yaml -f ingress_letsencrypt.yaml --version=2.10.1

4 - Check if the new certificate has indeed be issued

Use the following commands to check the renewal status:

kubectl get certificate,certificaterequest,order -n cattle-system

kubectl events -n cattle-system

Once the Rancher pod deployments have been rolled out, you should be able to log in to Rancher and verify the HTTPS certificate.

5 - Reconfigure Rancher agents to trust the private CA

As explained in Updating the Rancher certificate, 3 methods exist to force the Rancher agent to trust the CA. Method 3 can be used as a fallback if method 1 and 2 are not possible.

6 - Force Update Fleet clusters to reconnect the fleet-agent to Rancher

As explained in Updating the Rancher certificate, you have to select Force Update for the clusters within the Continuous Deliveryview of the Rancher UI to allow the fleet-agent in downstream clusters to successfully connect to Rancher.

7 - Remove private-key-rotation-policy annotation in ingress

You can now remove the annotation set in point 4 to prevent the renewal of the private key:

  • On the fly:
kubectl -n cattle-system annotate ingress rancher cert-manager.io/private-key-rotation-policy-
  • Permanently: Reapply the Rancher Helm chart removing the private-key-rotation-policy line in ingress_letsencrypt.yaml
> cat ingress_letsencrypt.yaml
ingress:
  tls:
    source: secret
  extraAnnotations:
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    cert-manager.io/private-key-algorithm: "ECDSA"
    cert-manager.io/private-key-size: "384"
privateCA: true

helm upgrade --install rancher rancher-prime/rancher --namespace cattle-system --set hostname=my_rancher.example.com -f values.yaml -f ingress_letsencrypt.yaml --version=2.10.1